May 1 21:58:20 oak-gw06 kernel: Initializing cgroup subsys cpuset May 1 21:58:20 oak-gw06 kernel: Initializing cgroup subsys cpu May 1 21:58:20 oak-gw06 kernel: Initializing cgroup subsys cpuacct May 1 21:58:20 oak-gw06 kernel: Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (sthiell@oak-rbh01) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Mon Mar 27 15:17:40 PDT 2017 May 1 21:58:20 oak-gw06 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=ad0ba2d1-f23b-47bd-bb0b-3cdfd331a9ad ro crashkernel=auto console=ttyS0,115200 LANG=en_US.UTF-8 May 1 21:58:20 oak-gw06 kernel: e820: BIOS-provided physical RAM map: May 1 21:58:20 oak-gw06 kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009f7ff] usable May 1 21:58:20 oak-gw06 kernel: BIOS-e820: [mem 0x000000000009f800-0x000000000009ffff] reserved May 1 21:58:20 oak-gw06 kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved May 1 21:58:20 oak-gw06 kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000bfffcfff] usable May 1 21:58:20 oak-gw06 kernel: BIOS-e820: [mem 0x00000000bfffd000-0x00000000bfffffff] reserved May 1 21:58:20 oak-gw06 kernel: BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved May 1 21:58:20 oak-gw06 kernel: BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved May 1 21:58:20 oak-gw06 kernel: BIOS-e820: [mem 0x0000000100000000-0x000000043fffffff] usable May 1 21:58:20 oak-gw06 kernel: NX (Execute Disable) protection: active May 1 21:58:20 oak-gw06 kernel: SMBIOS 2.4 present. May 1 21:58:20 oak-gw06 kernel: DMI: Red Hat KVM, BIOS 0.5.1 01/01/2011 May 1 21:58:20 oak-gw06 kernel: Hypervisor detected: KVM May 1 21:58:20 oak-gw06 kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved May 1 21:58:20 oak-gw06 kernel: e820: remove [mem 0x000a0000-0x000fffff] usable May 1 21:58:20 oak-gw06 kernel: e820: last_pfn = 0x440000 max_arch_pfn = 0x400000000 May 1 21:58:20 oak-gw06 kernel: MTRR default type: write-back May 1 21:58:20 oak-gw06 kernel: MTRR fixed ranges enabled: May 1 21:58:20 oak-gw06 kernel: 00000-9FFFF write-back May 1 21:58:20 oak-gw06 kernel: A0000-BFFFF uncachable May 1 21:58:20 oak-gw06 kernel: C0000-FFFFF write-protect May 1 21:58:20 oak-gw06 kernel: MTRR variable ranges enabled: May 1 21:58:20 oak-gw06 kernel: 0 base 0000C0000000 mask 3FFFC0000000 uncachable May 1 21:58:20 oak-gw06 kernel: 1 disabled May 1 21:58:20 oak-gw06 kernel: 2 disabled May 1 21:58:20 oak-gw06 kernel: 3 disabled May 1 21:58:20 oak-gw06 kernel: 4 disabled May 1 21:58:20 oak-gw06 kernel: 5 disabled May 1 21:58:20 oak-gw06 kernel: 6 disabled May 1 21:58:20 oak-gw06 kernel: 7 disabled May 1 21:58:20 oak-gw06 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 May 1 21:58:20 oak-gw06 kernel: e820: last_pfn = 0xbfffd max_arch_pfn = 0x400000000 May 1 21:58:20 oak-gw06 kernel: found SMP MP-table at [mem 0x000f7350-0x000f735f] mapped at [ffff8800000f7350] May 1 21:58:20 oak-gw06 kernel: Base memory trampoline at [ffff880000099000] 99000 size 24576 May 1 21:58:20 oak-gw06 kernel: Using GB pages for direct mapping May 1 21:58:20 oak-gw06 kernel: BRK [0x01f9e000, 0x01f9efff] PGTABLE May 1 21:58:20 oak-gw06 kernel: BRK [0x01f9f000, 0x01f9ffff] PGTABLE May 1 21:58:20 oak-gw06 kernel: BRK [0x01fa0000, 0x01fa0fff] PGTABLE May 1 21:58:20 oak-gw06 kernel: RAMDISK: [mem 0x35c36000-0x36e12fff] May 1 21:58:20 oak-gw06 kernel: ACPI: RSDP 00000000000f7300 00014 (v00 BOCHS ) May 1 21:58:20 oak-gw06 kernel: ACPI: RSDT 00000000bffffaba 00030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) May 1 21:58:20 oak-gw06 kernel: ACPI: FACP 00000000bfffeeb7 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) May 1 21:58:20 oak-gw06 kernel: ACPI: DSDT 00000000bfffdd80 01137 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001) May 1 21:58:20 oak-gw06 kernel: ACPI: FACS 00000000bfffdd40 00040 May 1 21:58:20 oak-gw06 kernel: ACPI: SSDT 00000000bfffef2b 00ADF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) May 1 21:58:20 oak-gw06 kernel: ACPI: APIC 00000000bffffa0a 000B0 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) May 1 21:58:20 oak-gw06 kernel: ACPI: Local APIC address 0xfee00000 May 1 21:58:20 oak-gw06 kernel: No NUMA configuration found May 1 21:58:20 oak-gw06 kernel: Faking a node at [mem 0x0000000000000000-0x000000043fffffff] May 1 21:58:20 oak-gw06 kernel: Initmem setup node 0 [mem 0x00000000-0x43fffffff] May 1 21:58:20 oak-gw06 kernel: NODE_DATA [mem 0x43ffd7000-0x43fffdfff] May 1 21:58:20 oak-gw06 kernel: Reserving 161MB of memory at 688MB for crashkernel (System RAM: 16383MB) May 1 21:58:20 oak-gw06 kernel: kvm-clock: Using msrs 4b564d01 and 4b564d00 May 1 21:58:20 oak-gw06 kernel: kvm-clock: cpu 0, msr 4:3ff87001, primary cpu clock May 1 21:58:20 oak-gw06 kernel: kvm-clock: using sched offset of 18958253217 cycles May 1 21:58:20 oak-gw06 kernel: Zone ranges: May 1 21:58:20 oak-gw06 kernel: DMA [mem 0x00001000-0x00ffffff] May 1 21:58:20 oak-gw06 kernel: DMA32 [mem 0x01000000-0xffffffff] May 1 21:58:20 oak-gw06 kernel: Normal [mem 0x100000000-0x43fffffff] May 1 21:58:20 oak-gw06 kernel: Movable zone start for each node May 1 21:58:20 oak-gw06 kernel: Early memory node ranges May 1 21:58:20 oak-gw06 kernel: node 0: [mem 0x00001000-0x0009efff] May 1 21:58:20 oak-gw06 kernel: node 0: [mem 0x00100000-0xbfffcfff] May 1 21:58:20 oak-gw06 kernel: node 0: [mem 0x100000000-0x43fffffff] May 1 21:58:20 oak-gw06 kernel: On node 0 totalpages: 4194203 May 1 21:58:20 oak-gw06 kernel: DMA zone: 64 pages used for memmap May 1 21:58:20 oak-gw06 kernel: DMA zone: 21 pages reserved May 1 21:58:20 oak-gw06 kernel: DMA zone: 3998 pages, LIFO batch:0 May 1 21:58:20 oak-gw06 kernel: DMA32 zone: 12224 pages used for memmap May 1 21:58:20 oak-gw06 kernel: DMA32 zone: 782333 pages, LIFO batch:31 May 1 21:58:20 oak-gw06 kernel: Normal zone: 53248 pages used for memmap May 1 21:58:20 oak-gw06 kernel: Normal zone: 3407872 pages, LIFO batch:31 May 1 21:58:20 oak-gw06 kernel: ACPI: PM-Timer IO Port: 0x608 May 1 21:58:20 oak-gw06 kernel: ACPI: Local APIC address 0xfee00000 May 1 21:58:20 oak-gw06 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) May 1 21:58:20 oak-gw06 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) May 1 21:58:20 oak-gw06 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) May 1 21:58:20 oak-gw06 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) May 1 21:58:20 oak-gw06 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled) May 1 21:58:20 oak-gw06 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled) May 1 21:58:20 oak-gw06 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) May 1 21:58:20 oak-gw06 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) May 1 21:58:20 oak-gw06 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) May 1 21:58:20 oak-gw06 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) May 1 21:58:20 oak-gw06 kernel: IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 May 1 21:58:20 oak-gw06 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) May 1 21:58:20 oak-gw06 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) May 1 21:58:20 oak-gw06 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) May 1 21:58:20 oak-gw06 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) May 1 21:58:20 oak-gw06 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) May 1 21:58:20 oak-gw06 kernel: ACPI: IRQ0 used by override. May 1 21:58:20 oak-gw06 kernel: ACPI: IRQ5 used by override. May 1 21:58:20 oak-gw06 kernel: ACPI: IRQ9 used by override. May 1 21:58:20 oak-gw06 kernel: ACPI: IRQ10 used by override. May 1 21:58:20 oak-gw06 kernel: ACPI: IRQ11 used by override. May 1 21:58:20 oak-gw06 kernel: Using ACPI (MADT) for SMP configuration information May 1 21:58:20 oak-gw06 kernel: smpboot: Allowing 8 CPUs, 0 hotplug CPUs May 1 21:58:20 oak-gw06 kernel: PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] May 1 21:58:20 oak-gw06 kernel: PM: Registered nosave memory: [mem 0x000a0000-0x000effff] May 1 21:58:20 oak-gw06 kernel: PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] May 1 21:58:20 oak-gw06 kernel: PM: Registered nosave memory: [mem 0xbfffd000-0xbfffffff] May 1 21:58:20 oak-gw06 kernel: PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] May 1 21:58:20 oak-gw06 kernel: PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] May 1 21:58:20 oak-gw06 kernel: PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] May 1 21:58:20 oak-gw06 kernel: PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] May 1 21:58:20 oak-gw06 kernel: e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices May 1 21:58:20 oak-gw06 kernel: Booting paravirtualized kernel on KVM May 1 21:58:20 oak-gw06 kernel: setup_percpu: NR_CPUS:5120 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1 May 1 21:58:20 oak-gw06 kernel: PERCPU: Embedded 33 pages/cpu @ffff88043fc00000 s96728 r8192 d30248 u262144 May 1 21:58:20 oak-gw06 kernel: pcpu-alloc: s96728 r8192 d30248 u262144 alloc=1*2097152 May 1 21:58:20 oak-gw06 kernel: pcpu-alloc: [0] 0 1 2 3 4 5 6 7 May 1 21:58:20 oak-gw06 kernel: KVM setup async PF for cpu 0 May 1 21:58:20 oak-gw06 kernel: kvm-stealtime: cpu 0, msr 43fc0f3c0 May 1 21:58:20 oak-gw06 kernel: Built 1 zonelists in Zone order, mobility grouping on. Total pages: 4128646 May 1 21:58:20 oak-gw06 kernel: Policy zone: Normal May 1 21:58:20 oak-gw06 kernel: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=ad0ba2d1-f23b-47bd-bb0b-3cdfd331a9ad ro crashkernel=auto console=ttyS0,115200 LANG=en_US.UTF-8 May 1 21:58:20 oak-gw06 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes) May 1 21:58:20 oak-gw06 kernel: x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 May 1 21:58:20 oak-gw06 kernel: xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form May 1 21:58:20 oak-gw06 kernel: Memory: 4977652k/17825792k available (6765k kernel code, 1048980k absent, 529252k reserved, 4431k data, 1680k init) May 1 21:58:20 oak-gw06 kernel: SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1 May 1 21:58:20 oak-gw06 kernel: Hierarchical RCU implementation. May 1 21:58:20 oak-gw06 kernel: #011RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=8. May 1 21:58:20 oak-gw06 kernel: NR_IRQS:327936 nr_irqs:488 0 May 1 21:58:20 oak-gw06 kernel: Console: colour VGA+ 80x25 May 1 21:58:20 oak-gw06 kernel: console [ttyS0] enabled May 1 21:58:20 oak-gw06 kernel: allocated 67108864 bytes of page_cgroup May 1 21:58:20 oak-gw06 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups May 1 21:58:20 oak-gw06 kernel: tsc: Detected 2299.996 MHz processor May 1 21:58:20 oak-gw06 kernel: Calibrating delay loop (skipped) preset value.. 4599.99 BogoMIPS (lpj=2299996) May 1 21:58:20 oak-gw06 kernel: pid_max: default: 32768 minimum: 301 May 1 21:58:20 oak-gw06 kernel: Security Framework initialized May 1 21:58:20 oak-gw06 kernel: SELinux: Initializing. May 1 21:58:20 oak-gw06 kernel: SELinux: Starting in permissive mode May 1 21:58:20 oak-gw06 kernel: Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes) May 1 21:58:20 oak-gw06 kernel: Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes) May 1 21:58:20 oak-gw06 kernel: Mount-cache hash table entries: 4096 May 1 21:58:20 oak-gw06 kernel: Initializing cgroup subsys memory May 1 21:58:20 oak-gw06 kernel: Initializing cgroup subsys devices May 1 21:58:20 oak-gw06 kernel: Initializing cgroup subsys freezer May 1 21:58:20 oak-gw06 kernel: Initializing cgroup subsys net_cls May 1 21:58:20 oak-gw06 kernel: Initializing cgroup subsys blkio May 1 21:58:20 oak-gw06 kernel: Initializing cgroup subsys perf_event May 1 21:58:20 oak-gw06 kernel: Initializing cgroup subsys hugetlb May 1 21:58:20 oak-gw06 kernel: Initializing cgroup subsys pids May 1 21:58:20 oak-gw06 kernel: Initializing cgroup subsys net_prio May 1 21:58:20 oak-gw06 kernel: mce: CPU supports 10 MCE banks May 1 21:58:20 oak-gw06 kernel: Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 May 1 21:58:20 oak-gw06 kernel: Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 May 1 21:58:20 oak-gw06 kernel: tlb_flushall_shift: 6 May 1 21:58:20 oak-gw06 kernel: Freeing SMP alternatives: 28k freed May 1 21:58:20 oak-gw06 kernel: ACPI: Core revision 20130517 May 1 21:58:20 oak-gw06 kernel: ACPI: All ACPI Tables successfully acquired May 1 21:58:20 oak-gw06 kernel: ftrace: allocating 25813 entries in 101 pages May 1 21:58:20 oak-gw06 kernel: smpboot: APIC(0) Converting physical 0 to logical package 0 May 1 21:58:20 oak-gw06 kernel: smpboot: APIC(1) Converting physical 1 to logical package 1 May 1 21:58:20 oak-gw06 kernel: smpboot: APIC(2) Converting physical 2 to logical package 2 May 1 21:58:20 oak-gw06 kernel: smpboot: APIC(3) Converting physical 3 to logical package 3 May 1 21:58:20 oak-gw06 kernel: smpboot: APIC(4) Converting physical 4 to logical package 4 May 1 21:58:20 oak-gw06 kernel: smpboot: APIC(5) Converting physical 5 to logical package 5 May 1 21:58:20 oak-gw06 kernel: smpboot: APIC(6) Converting physical 6 to logical package 6 May 1 21:58:20 oak-gw06 kernel: smpboot: APIC(7) Converting physical 7 to logical package 7 May 1 21:58:20 oak-gw06 kernel: smpboot: Max logical packages: 8 May 1 21:58:20 oak-gw06 kernel: Enabling x2apic May 1 21:58:20 oak-gw06 kernel: Enabled x2apic May 1 21:58:20 oak-gw06 kernel: Switched APIC routing to physical x2apic. May 1 21:58:20 oak-gw06 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 May 1 21:58:20 oak-gw06 kernel: smpboot: CPU0: Intel Core Processor (Haswell) (fam: 06, model: 3c, stepping: 01) May 1 21:58:20 oak-gw06 kernel: TSC deadline timer enabled May 1 21:58:20 oak-gw06 kernel: Performance Events: unsupported p6 CPU model 60 no PMU driver, software events only. May 1 21:58:20 oak-gw06 kernel: KVM setup paravirtual spinlock May 1 21:58:20 oak-gw06 kernel: kvm-clock: cpu 1, msr 4:3ff87041, secondary cpu clock May 1 21:58:20 oak-gw06 kernel: KVM setup async PF for cpu 1 May 1 21:58:20 oak-gw06 kernel: kvm-clock: cpu 2, msr 4:3ff87081, secondary cpu clock May 1 21:58:20 oak-gw06 kernel: kvm-stealtime: cpu 1, msr 43fc4f3c0 May 1 21:58:20 oak-gw06 kernel: KVM setup async PF for cpu 2 May 1 21:58:20 oak-gw06 kernel: kvm-clock: cpu 3, msr 4:3ff870c1, secondary cpu clock May 1 21:58:20 oak-gw06 kernel: kvm-stealtime: cpu 2, msr 43fc8f3c0 May 1 21:58:20 oak-gw06 kernel: KVM setup async PF for cpu 3 May 1 21:58:20 oak-gw06 kernel: kvm-clock: cpu 4, msr 4:3ff87101, secondary cpu clock May 1 21:58:20 oak-gw06 kernel: kvm-stealtime: cpu 3, msr 43fccf3c0 May 1 21:58:20 oak-gw06 kernel: KVM setup async PF for cpu 4 May 1 21:58:20 oak-gw06 kernel: kvm-clock: cpu 5, msr 4:3ff87141, secondary cpu clock May 1 21:58:20 oak-gw06 kernel: kvm-stealtime: cpu 4, msr 43fd0f3c0 May 1 21:58:20 oak-gw06 kernel: KVM setup async PF for cpu 5 May 1 21:58:20 oak-gw06 kernel: kvm-clock: cpu 6, msr 4:3ff87181, secondary cpu clock May 1 21:58:20 oak-gw06 kernel: kvm-stealtime: cpu 5, msr 43fd4f3c0 May 1 21:58:20 oak-gw06 kernel: KVM setup async PF for cpu 6 May 1 21:58:20 oak-gw06 kernel: smpboot: Booting Node 0, Processors #1 #2 #3 #4 #5 #6 #7 OK May 1 21:58:20 oak-gw06 kernel: kvm-clock: cpu 7, msr 4:3ff871c1, secondary cpu clock May 1 21:58:20 oak-gw06 kernel: kvm-stealtime: cpu 6, msr 43fd8f3c0 May 1 21:58:20 oak-gw06 kernel: Brought up 8 CPUs May 1 21:58:20 oak-gw06 kernel: KVM setup async PF for cpu 7 May 1 21:58:20 oak-gw06 kernel: kvm-stealtime: cpu 7, msr 43fdcf3c0 May 1 21:58:20 oak-gw06 kernel: smpboot: Total of 8 processors activated (36799.93 BogoMIPS) May 1 21:58:20 oak-gw06 kernel: node 0 initialised, 2817477 pages in 51ms May 1 21:58:20 oak-gw06 kernel: devtmpfs: initialized May 1 21:58:20 oak-gw06 kernel: EVM: security.selinux May 1 21:58:20 oak-gw06 kernel: EVM: security.ima May 1 21:58:20 oak-gw06 kernel: EVM: security.capability May 1 21:58:20 oak-gw06 kernel: atomic64 test passed for x86-64 platform with CX8 and with SSE May 1 21:58:20 oak-gw06 kernel: pinctrl core: initialized pinctrl subsystem May 1 21:58:20 oak-gw06 kernel: NET: Registered protocol family 16 May 1 21:58:20 oak-gw06 kernel: ACPI: bus type PCI registered May 1 21:58:20 oak-gw06 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 May 1 21:58:20 oak-gw06 kernel: PCI: Using configuration type 1 for base access May 1 21:58:20 oak-gw06 kernel: ACPI: Added _OSI(Module Device) May 1 21:58:20 oak-gw06 kernel: ACPI: Added _OSI(Processor Device) May 1 21:58:20 oak-gw06 kernel: ACPI: Added _OSI(3.0 _SCP Extensions) May 1 21:58:20 oak-gw06 kernel: ACPI: Added _OSI(Processor Aggregator Device) May 1 21:58:20 oak-gw06 kernel: ACPI: EC: Look up EC in DSDT May 1 21:58:20 oak-gw06 kernel: ACPI: Interpreter enabled May 1 21:58:20 oak-gw06 kernel: ACPI: (supports S0 S5) May 1 21:58:20 oak-gw06 kernel: ACPI: Using IOAPIC for interrupt routing May 1 21:58:20 oak-gw06 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug May 1 21:58:20 oak-gw06 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) May 1 21:58:20 oak-gw06 kernel: acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] May 1 21:58:20 oak-gw06 kernel: acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM May 1 21:58:20 oak-gw06 kernel: acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [3] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [4] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [5] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [6] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [7] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [8] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [9] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [10] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [11] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [12] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [13] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [14] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [15] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [16] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [17] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [18] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [19] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [20] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [21] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [22] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [23] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [24] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [25] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [26] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [27] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [28] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [29] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [30] registered May 1 21:58:20 oak-gw06 kernel: acpiphp: Slot [31] registered May 1 21:58:20 oak-gw06 kernel: PCI host bridge to bus 0000:00 May 1 21:58:20 oak-gw06 kernel: pci_bus 0000:00: root bus resource [bus 00-ff] May 1 21:58:20 oak-gw06 kernel: pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] May 1 21:58:20 oak-gw06 kernel: pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] May 1 21:58:20 oak-gw06 kernel: pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] May 1 21:58:20 oak-gw06 kernel: pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 May 1 21:58:20 oak-gw06 kernel: pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 May 1 21:58:20 oak-gw06 kernel: pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 May 1 21:58:20 oak-gw06 kernel: pci 0000:00:01.1: reg 0x20: [io 0xc060-0xc06f] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:01.2: [8086:7020] type 00 class 0x0c0300 May 1 21:58:20 oak-gw06 kernel: pci 0000:00:01.2: reg 0x20: [io 0xc000-0xc01f] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 May 1 21:58:20 oak-gw06 kernel: pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI May 1 21:58:20 oak-gw06 kernel: pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB May 1 21:58:20 oak-gw06 kernel: pci 0000:00:02.0: [1234:1111] type 00 class 0x030000 May 1 21:58:20 oak-gw06 kernel: pci 0000:00:02.0: reg 0x10: [mem 0xfd800000-0xfdffffff pref] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:02.0: reg 0x18: [mem 0xfebd0000-0xfebd0fff] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:02.0: reg 0x30: [mem 0xfebc0000-0xfebcffff pref] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000 May 1 21:58:20 oak-gw06 kernel: pci 0000:00:03.0: reg 0x10: [io 0xc020-0xc03f] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:03.0: reg 0x14: [mem 0xfebd1000-0xfebd1fff] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:03.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:05.0: [1af4:1002] type 00 class 0x00ff00 May 1 21:58:20 oak-gw06 kernel: pci 0000:00:05.0: reg 0x10: [io 0xc040-0xc05f] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:06.0: [14e4:16a9] type 00 class 0x020000 May 1 21:58:20 oak-gw06 kernel: pci 0000:00:06.0: reg 0x10: [mem 0xfe800000-0xfe807fff 64bit pref] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:06.0: reg 0x20: [mem 0xfe808000-0xfe809fff 64bit pref] May 1 21:58:20 oak-gw06 kernel: pci 0000:00:07.0: [15b3:1004] type 00 class 0x028000 May 1 21:58:20 oak-gw06 kernel: pci 0000:00:07.0: reg 0x18: [mem 0xfe000000-0xfe7fffff 64bit pref] May 1 21:58:20 oak-gw06 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) May 1 21:58:20 oak-gw06 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) May 1 21:58:20 oak-gw06 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) May 1 21:58:20 oak-gw06 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) May 1 21:58:20 oak-gw06 kernel: ACPI: PCI Interrupt Link [LNKS] (IRQs *9) May 1 21:58:20 oak-gw06 kernel: ACPI: Enabled 16 GPEs in block 00 to 0F May 1 21:58:20 oak-gw06 kernel: vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none May 1 21:58:20 oak-gw06 kernel: vgaarb: loaded May 1 21:58:20 oak-gw06 kernel: vgaarb: bridge control possible 0000:00:02.0 May 1 21:58:20 oak-gw06 kernel: SCSI subsystem initialized May 1 21:58:20 oak-gw06 kernel: ACPI: bus type USB registered May 1 21:58:20 oak-gw06 kernel: usbcore: registered new interface driver usbfs May 1 21:58:20 oak-gw06 kernel: usbcore: registered new interface driver hub May 1 21:58:20 oak-gw06 kernel: usbcore: registered new device driver usb May 1 21:58:20 oak-gw06 kernel: PCI: Using ACPI for IRQ routing May 1 21:58:20 oak-gw06 kernel: PCI: pci_cache_line_size set to 64 bytes May 1 21:58:20 oak-gw06 kernel: e820: reserve RAM buffer [mem 0x0009f800-0x0009ffff] May 1 21:58:20 oak-gw06 kernel: e820: reserve RAM buffer [mem 0xbfffd000-0xbfffffff] May 1 21:58:20 oak-gw06 kernel: NetLabel: Initializing May 1 21:58:20 oak-gw06 kernel: NetLabel: domain hash size = 128 May 1 21:58:20 oak-gw06 kernel: NetLabel: protocols = UNLABELED CIPSOv4 May 1 21:58:20 oak-gw06 kernel: NetLabel: unlabeled traffic allowed by default May 1 21:58:20 oak-gw06 kernel: Switched to clocksource kvm-clock May 1 21:58:20 oak-gw06 kernel: pnp: PnP ACPI init May 1 21:58:20 oak-gw06 kernel: ACPI: bus type PNP registered May 1 21:58:20 oak-gw06 kernel: pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active) May 1 21:58:20 oak-gw06 kernel: pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active) May 1 21:58:20 oak-gw06 kernel: pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active) May 1 21:58:20 oak-gw06 kernel: pnp 00:03: [dma 2] May 1 21:58:20 oak-gw06 kernel: pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active) May 1 21:58:20 oak-gw06 kernel: pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) May 1 21:58:20 oak-gw06 kernel: pnp: PnP ACPI: found 5 devices May 1 21:58:20 oak-gw06 kernel: ACPI: bus type PNP unregistered May 1 21:58:20 oak-gw06 kernel: pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] May 1 21:58:20 oak-gw06 kernel: pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] May 1 21:58:20 oak-gw06 kernel: pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] May 1 21:58:20 oak-gw06 kernel: pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] May 1 21:58:20 oak-gw06 kernel: NET: Registered protocol family 2 May 1 21:58:20 oak-gw06 kernel: TCP established hash table entries: 131072 (order: 8, 1048576 bytes) May 1 21:58:20 oak-gw06 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) May 1 21:58:20 oak-gw06 kernel: TCP: Hash tables configured (established 131072 bind 65536) May 1 21:58:20 oak-gw06 kernel: TCP: reno registered May 1 21:58:20 oak-gw06 kernel: UDP hash table entries: 8192 (order: 6, 262144 bytes) May 1 21:58:20 oak-gw06 kernel: UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes) May 1 21:58:20 oak-gw06 kernel: NET: Registered protocol family 1 May 1 21:58:20 oak-gw06 kernel: pci 0000:00:00.0: Limiting direct PCI/PCI transfers May 1 21:58:20 oak-gw06 kernel: pci 0000:00:01.0: PIIX3: Enabling Passive Release May 1 21:58:20 oak-gw06 kernel: pci 0000:00:01.0: Activating ISA DMA hang workarounds May 1 21:58:20 oak-gw06 kernel: ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 May 1 21:58:20 oak-gw06 kernel: pci 0000:00:02.0: Boot video device May 1 21:58:20 oak-gw06 kernel: PCI: CLS 0 bytes, default 64 May 1 21:58:20 oak-gw06 kernel: Unpacking initramfs... May 1 21:58:20 oak-gw06 kernel: Freeing initrd memory: 18292k freed May 1 21:58:20 oak-gw06 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB) May 1 21:58:20 oak-gw06 kernel: software IO TLB [mem 0xbbffd000-0xbfffd000] (64MB) mapped at [ffff8800bbffd000-ffff8800bfffcfff] May 1 21:58:20 oak-gw06 kernel: sha1_ssse3: Using AVX2 optimized SHA-1 implementation May 1 21:58:20 oak-gw06 kernel: sha256_ssse3: Using AVX2 optimized SHA-256 implementation May 1 21:58:20 oak-gw06 kernel: futex hash table entries: 2048 (order: 5, 131072 bytes) May 1 21:58:20 oak-gw06 kernel: Initialise system trusted keyring May 1 21:58:20 oak-gw06 kernel: audit: initializing netlink socket (disabled) May 1 21:58:20 oak-gw06 kernel: type=2000 audit(1493701100.838:1): initialized May 1 21:58:20 oak-gw06 kernel: HugeTLB registered 1 GB page size, pre-allocated 0 pages May 1 21:58:20 oak-gw06 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages May 1 21:58:20 oak-gw06 kernel: zpool: loaded May 1 21:58:20 oak-gw06 kernel: zbud: loaded May 1 21:58:20 oak-gw06 kernel: VFS: Disk quotas dquot_6.5.2 May 1 21:58:20 oak-gw06 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) May 1 21:58:20 oak-gw06 kernel: msgmni has been set to 31769 May 1 21:58:20 oak-gw06 kernel: Key type big_key registered May 1 21:58:20 oak-gw06 kernel: SELinux: Registering netfilter hooks May 1 21:58:20 oak-gw06 kernel: NET: Registered protocol family 38 May 1 21:58:20 oak-gw06 kernel: Key type asymmetric registered May 1 21:58:20 oak-gw06 kernel: Asymmetric key parser 'x509' registered May 1 21:58:20 oak-gw06 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) May 1 21:58:20 oak-gw06 kernel: io scheduler noop registered May 1 21:58:20 oak-gw06 kernel: io scheduler deadline registered (default) May 1 21:58:20 oak-gw06 kernel: io scheduler cfq registered May 1 21:58:20 oak-gw06 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 May 1 21:58:20 oak-gw06 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4 May 1 21:58:20 oak-gw06 kernel: intel_idle: does not run on family 6 model 60 May 1 21:58:20 oak-gw06 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 May 1 21:58:20 oak-gw06 kernel: ACPI: Power Button [PWRF] May 1 21:58:20 oak-gw06 kernel: GHES: HEST is not enabled! May 1 21:58:20 oak-gw06 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled May 1 21:58:20 oak-gw06 kernel: 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A May 1 21:58:20 oak-gw06 kernel: Non-volatile memory driver v1.3 May 1 21:58:20 oak-gw06 kernel: Linux agpgart interface v0.103 May 1 21:58:20 oak-gw06 kernel: crash memory driver: version 1.1 May 1 21:58:20 oak-gw06 kernel: rdac: device handler registered May 1 21:58:20 oak-gw06 kernel: hp_sw: device handler registered May 1 21:58:20 oak-gw06 kernel: emc: device handler registered May 1 21:58:20 oak-gw06 kernel: alua: device handler registered May 1 21:58:20 oak-gw06 kernel: libphy: Fixed MDIO Bus: probed May 1 21:58:20 oak-gw06 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver May 1 21:58:20 oak-gw06 kernel: ehci-pci: EHCI PCI platform driver May 1 21:58:20 oak-gw06 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver May 1 21:58:20 oak-gw06 kernel: ohci-pci: OHCI PCI platform driver May 1 21:58:20 oak-gw06 kernel: uhci_hcd: USB Universal Host Controller Interface driver May 1 21:58:20 oak-gw06 kernel: uhci_hcd 0000:00:01.2: UHCI Host Controller May 1 21:58:20 oak-gw06 kernel: uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 May 1 21:58:20 oak-gw06 kernel: uhci_hcd 0000:00:01.2: detected 2 ports May 1 21:58:20 oak-gw06 kernel: uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c000 May 1 21:58:20 oak-gw06 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 May 1 21:58:20 oak-gw06 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 May 1 21:58:20 oak-gw06 kernel: usb usb1: Product: UHCI Host Controller May 1 21:58:20 oak-gw06 kernel: usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd May 1 21:58:20 oak-gw06 kernel: usb usb1: SerialNumber: 0000:00:01.2 May 1 21:58:20 oak-gw06 kernel: hub 1-0:1.0: USB hub found May 1 21:58:20 oak-gw06 kernel: hub 1-0:1.0: 2 ports detected May 1 21:58:20 oak-gw06 kernel: usbcore: registered new interface driver usbserial May 1 21:58:20 oak-gw06 kernel: usbcore: registered new interface driver usbserial_generic May 1 21:58:20 oak-gw06 kernel: usbserial: USB Serial support registered for generic May 1 21:58:20 oak-gw06 kernel: i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 May 1 21:58:20 oak-gw06 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1 May 1 21:58:20 oak-gw06 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12 May 1 21:58:20 oak-gw06 kernel: mousedev: PS/2 mouse device common for all mice May 1 21:58:20 oak-gw06 kernel: input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 May 1 21:58:20 oak-gw06 kernel: rtc_cmos 00:00: RTC can wake from S4 May 1 21:58:20 oak-gw06 kernel: input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 May 1 21:58:20 oak-gw06 kernel: input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 May 1 21:58:20 oak-gw06 kernel: rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 May 1 21:58:20 oak-gw06 kernel: rtc_cmos 00:00: alarms up to one day, 114 bytes nvram May 1 21:58:20 oak-gw06 kernel: cpuidle: using governor menu May 1 21:58:20 oak-gw06 kernel: hidraw: raw HID events driver (C) Jiri Kosina May 1 21:58:20 oak-gw06 kernel: usbcore: registered new interface driver usbhid May 1 21:58:20 oak-gw06 kernel: usbhid: USB HID core driver May 1 21:58:20 oak-gw06 kernel: drop_monitor: Initializing network drop monitor service May 1 21:58:20 oak-gw06 kernel: TCP: cubic registered May 1 21:58:20 oak-gw06 kernel: Initializing XFRM netlink socket May 1 21:58:20 oak-gw06 kernel: NET: Registered protocol family 10 May 1 21:58:20 oak-gw06 kernel: NET: Registered protocol family 17 May 1 21:58:20 oak-gw06 kernel: microcode: CPU0 sig=0x306c1, pf=0x1, revision=0x1 May 1 21:58:20 oak-gw06 kernel: microcode: CPU1 sig=0x306c1, pf=0x1, revision=0x1 May 1 21:58:20 oak-gw06 kernel: microcode: CPU2 sig=0x306c1, pf=0x1, revision=0x1 May 1 21:58:20 oak-gw06 kernel: microcode: CPU3 sig=0x306c1, pf=0x1, revision=0x1 May 1 21:58:20 oak-gw06 kernel: microcode: CPU4 sig=0x306c1, pf=0x1, revision=0x1 May 1 21:58:20 oak-gw06 kernel: microcode: CPU5 sig=0x306c1, pf=0x1, revision=0x1 May 1 21:58:20 oak-gw06 kernel: microcode: CPU6 sig=0x306c1, pf=0x1, revision=0x1 May 1 21:58:20 oak-gw06 kernel: microcode: CPU7 sig=0x306c1, pf=0x1, revision=0x1 May 1 21:58:20 oak-gw06 kernel: microcode: Microcode Update Driver: v2.01 , Peter Oruba May 1 21:58:20 oak-gw06 kernel: Loading compiled-in X.509 certificates May 1 21:58:20 oak-gw06 kernel: Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' May 1 21:58:20 oak-gw06 kernel: Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' May 1 21:58:20 oak-gw06 kernel: Loaded X.509 cert 'CentOS Linux kernel signing key: ff557397f0535e4a97ca7851e2eaf869307bd012' May 1 21:58:20 oak-gw06 kernel: registered taskstats version 1 May 1 21:58:20 oak-gw06 kernel: Key type trusted registered May 1 21:58:20 oak-gw06 kernel: Key type encrypted registered May 1 21:58:20 oak-gw06 kernel: IMA: No TPM chip found, activating TPM-bypass! May 1 21:58:20 oak-gw06 kernel: rtc_cmos 00:00: setting system clock to 2017-05-02 04:58:20 UTC (1493701100) May 1 21:58:20 oak-gw06 kernel: Freeing unused kernel memory: 1680k freed May 1 21:58:20 oak-gw06 systemd[1]: Starting Create list of required static device nodes for the current kernel... May 1 21:58:20 oak-gw06 kernel: usb 1-1: new full-speed USB device number 2 using uhci_hcd May 1 21:58:20 oak-gw06 systemd: Started Create list of required static device nodes for the current kernel. May 1 21:58:20 oak-gw06 kernel: pps_core: LinuxPPS API ver. 1 registered May 1 21:58:20 oak-gw06 kernel: libata version 3.00 loaded. May 1 21:58:20 oak-gw06 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti May 1 21:58:20 oak-gw06 kernel: FDC 0 is a S82078B May 1 21:58:20 oak-gw06 kernel: ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 May 1 21:58:20 oak-gw06 kernel: virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver May 1 21:58:20 oak-gw06 kernel: ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 May 1 21:58:20 oak-gw06 kernel: virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver May 1 21:58:20 oak-gw06 kernel: ata_piix 0000:00:01.1: version 2.13 May 1 21:58:20 oak-gw06 kernel: [drm] Initialized drm 1.1.0 20060810 May 1 21:58:20 oak-gw06 kernel: scsi host0: ata_piix May 1 21:58:20 oak-gw06 kernel: scsi host1: ata_piix May 1 21:58:20 oak-gw06 kernel: ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc060 irq 14 May 1 21:58:20 oak-gw06 kernel: ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc068 irq 15 May 1 21:58:20 oak-gw06 kernel: virtio-pci 0000:00:03.0: irq 24 for MSI/MSI-X May 1 21:58:20 oak-gw06 kernel: virtio-pci 0000:00:03.0: irq 25 for MSI/MSI-X May 1 21:58:20 oak-gw06 kernel: virtio-pci 0000:00:03.0: irq 26 for MSI/MSI-X May 1 21:58:20 oak-gw06 kernel: PTP clock support registered May 1 21:58:20 oak-gw06 kernel: mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014) May 1 21:58:20 oak-gw06 kernel: mlx4_core: Initializing 0000:00:07.0 May 1 21:58:20 oak-gw06 kernel: mlx4_core 0000:00:07.0: Detected virtual function - running in slave mode May 1 21:58:20 oak-gw06 kernel: mlx4_core 0000:00:07.0: Sending reset May 1 21:58:20 oak-gw06 kernel: bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.712.30-0 (2014/02/10) May 1 21:58:20 oak-gw06 kernel: usb 1-1: New USB device found, idVendor=0627, idProduct=0001 May 1 21:58:20 oak-gw06 kernel: usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5 May 1 21:58:20 oak-gw06 kernel: usb 1-1: Product: QEMU USB Tablet May 1 21:58:20 oak-gw06 kernel: usb 1-1: Manufacturer: QEMU May 1 21:58:20 oak-gw06 kernel: usb 1-1: SerialNumber: 42 May 1 21:58:20 oak-gw06 kernel: input: QEMU QEMU USB Tablet as /devices/pci0000:00/0000:00:01.2/usb1/1-1/1-1:1.0/input/input4 May 1 21:58:20 oak-gw06 kernel: hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v0.01 Pointer [QEMU QEMU USB Tablet] on usb-0000:00:01.2-1/input0 May 1 21:58:20 oak-gw06 kernel: mlx4_core 0000:00:07.0: Sending vhcr0 May 1 21:58:20 oak-gw06 kernel: mlx4_core 0000:00:07.0: HCA minimum page size:512 May 1 21:58:20 oak-gw06 kernel: mlx4_core 0000:00:07.0: Timestamping is not supported in slave mode May 1 21:58:20 oak-gw06 kernel: mlx4_core 0000:00:07.0: irq 27 for MSI/MSI-X May 1 21:58:20 oak-gw06 kernel: mlx4_core 0000:00:07.0: irq 28 for MSI/MSI-X May 1 21:58:20 oak-gw06 kernel: mlx4_core 0000:00:07.0: irq 29 for MSI/MSI-X May 1 21:58:20 oak-gw06 kernel: mlx4_core 0000:00:07.0: irq 30 for MSI/MSI-X May 1 21:58:20 oak-gw06 kernel: mlx4_core 0000:00:07.0: irq 31 for MSI/MSI-X May 1 21:58:20 oak-gw06 kernel: mlx4_core 0000:00:07.0: irq 32 for MSI/MSI-X May 1 21:58:20 oak-gw06 kernel: mlx4_core 0000:00:07.0: irq 33 for MSI/MSI-X May 1 21:58:20 oak-gw06 kernel: mlx4_core 0000:00:07.0: irq 34 for MSI/MSI-X May 1 21:58:20 oak-gw06 kernel: mlx4_core 0000:00:07.0: irq 35 for MSI/MSI-X May 1 21:58:20 oak-gw06 kernel: bnx2x 0000:00:06.0: msix capability found May 1 21:58:20 oak-gw06 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014) May 1 21:58:20 oak-gw06 kernel: ata1.00: ATA-7: QEMU HARDDISK, 1.5.3, max UDMA/100 May 1 21:58:20 oak-gw06 kernel: ata1.00: 41943040 sectors, multi 16: LBA48 May 1 21:58:20 oak-gw06 kernel: ata1.01: ATAPI: QEMU DVD-ROM, 1.5.3, max UDMA/100 May 1 21:58:20 oak-gw06 kernel: ata1.00: configured for MWDMA2 May 1 21:58:20 oak-gw06 kernel: ata1.01: configured for MWDMA2 May 1 21:58:20 oak-gw06 kernel: scsi 0:0:0:0: Direct-Access ATA QEMU HARDDISK 3 PQ: 0 ANSI: 5 May 1 21:58:20 oak-gw06 kernel: scsi 0:0:1:0: CD-ROM QEMU QEMU DVD-ROM 1.5. PQ: 0 ANSI: 5 May 1 21:58:21 oak-gw06 kernel: bnx2x 0000:00:06.0: irq 36 for MSI/MSI-X May 1 21:58:21 oak-gw06 kernel: bnx2x 0000:00:06.0: irq 37 for MSI/MSI-X May 1 21:58:21 oak-gw06 kernel: [drm] Found bochs VGA, ID 0xb0c0. May 1 21:58:21 oak-gw06 kernel: [drm] Framebuffer size 8192 kB @ 0xfd800000, mmio @ 0xfebd0000. May 1 21:58:21 oak-gw06 kernel: [TTM] Zone kernel: Available graphics memory: 8133780 kiB May 1 21:58:21 oak-gw06 kernel: [TTM] Zone dma32: Available graphics memory: 2097152 kiB May 1 21:58:21 oak-gw06 kernel: [TTM] Initializing pool allocator May 1 21:58:21 oak-gw06 kernel: [TTM] Initializing DMA pool allocator May 1 21:58:21 oak-gw06 kernel: fbcon: bochsdrmfb (fb0) is primary device May 1 21:58:21 oak-gw06 kernel: Console: switching to colour frame buffer device 128x48 May 1 21:58:21 oak-gw06 kernel: bochs-drm 0000:00:02.0: fb0: bochsdrmfb frame buffer device May 1 21:58:21 oak-gw06 kernel: [drm] Initialized bochs-drm 1.0.0 20130925 for 0000:00:02.0 on minor 0 May 1 21:58:21 oak-gw06 kernel: sr 0:0:1:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray May 1 21:58:21 oak-gw06 kernel: sd 0:0:0:0: [sda] 41943040 512-byte logical blocks: (21.4 GB/20.0 GiB) May 1 21:58:21 oak-gw06 kernel: sd 0:0:0:0: [sda] Write Protect is off May 1 21:58:21 oak-gw06 kernel: sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 May 1 21:58:21 oak-gw06 kernel: sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA May 1 21:58:21 oak-gw06 kernel: cdrom: Uniform CD-ROM driver Revision: 3.20 May 1 21:58:21 oak-gw06 kernel: sda: sda1 sda2 May 1 21:58:21 oak-gw06 kernel: sr 0:0:1:0: Attached scsi CD-ROM sr0 May 1 21:58:21 oak-gw06 kernel: sd 0:0:0:0: [sda] Attached SCSI disk May 1 21:58:21 oak-gw06 kernel: tsc: Refined TSC clocksource calibration: 2299.997 MHz May 1 21:58:21 oak-gw06 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null) May 1 21:58:21 oak-gw06 systemd: Stopped Create list of required static device nodes for the current kernel. May 1 21:58:21 oak-gw06 systemd: Stopping Create list of required static device nodes for the current kernel... May 1 21:58:23 oak-gw06 kernel: SELinux: Disabled at runtime. May 1 21:58:23 oak-gw06 kernel: SELinux: Unregistering netfilter hooks May 1 21:58:23 oak-gw06 kernel: type=1404 audit(1493701102.044:2): selinux=0 auid=4294967295 ses=4294967295 May 1 21:58:23 oak-gw06 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team May 1 21:58:23 oak-gw06 kernel: EXT4-fs (sda1): re-mounted. Opts: (null) May 1 21:58:23 oak-gw06 kernel: TCP: htcp registered May 1 21:58:23 oak-gw06 kernel: RPC: Registered named UNIX socket transport module. May 1 21:58:23 oak-gw06 kernel: RPC: Registered udp transport module. May 1 21:58:23 oak-gw06 kernel: RPC: Registered tcp transport module. May 1 21:58:23 oak-gw06 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module. May 1 21:58:23 oak-gw06 systemd: Binding to IPv6 address not available since kernel does not support IPv6. May 1 21:58:23 oak-gw06 systemd: Started Create list of required static device nodes for the current kernel. May 1 21:58:23 oak-gw06 kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de). May 1 21:58:23 oak-gw06 systemd: Starting Initialize the iWARP/InfiniBand/RDMA stack in the kernel... May 1 21:58:24 oak-gw06 kernel: input: PC Speaker as /devices/platform/pcspkr/input/input5 May 1 21:58:24 oak-gw06 kernel: piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 May 1 21:58:24 oak-gw06 kernel: ppdev: user-space parallel port driver May 1 21:58:24 oak-gw06 kernel: AES CTR mode by8 optimization enabled May 1 21:58:24 oak-gw06 kernel: alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) May 1 21:58:24 oak-gw06 kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0 May 1 21:58:24 oak-gw06 kernel: sr 0:0:1:0: Attached scsi generic sg1 type 5 May 1 21:58:24 oak-gw06 kernel: alg: No test for crc32 (crc32-pclmul) May 1 21:58:24 oak-gw06 kernel: intel_rapl: no valid rapl domains found in package 0 May 1 21:58:25 oak-gw06 kernel: intel_rapl: no valid rapl domains found in package 0 May 1 21:58:25 oak-gw06 kernel: intel_rapl: no valid rapl domains found in package 0 May 1 21:58:25 oak-gw06 kernel: intel_rapl: no valid rapl domains found in package 0 May 1 21:58:25 oak-gw06 kernel: intel_rapl: no valid rapl domains found in package 0 May 1 21:58:25 oak-gw06 kernel: intel_rapl: no valid rapl domains found in package 0 May 1 21:58:25 oak-gw06 kernel: intel_rapl: no valid rapl domains found in package 0 May 1 21:58:25 oak-gw06 kernel: intel_powerclamp: No package C-state available May 1 21:58:25 oak-gw06 kernel: intel_powerclamp: No package C-state available May 1 21:58:25 oak-gw06 kernel: type=1305 audit(1493701105.416:3): audit_pid=517 old=0 auid=4294967295 ses=4294967295 res=1 May 1 21:58:25 oak-gw06 kernel: Adding 4194300k swap on /dev/sda2. Priority:-1 extents:1 across:4194300k FS May 1 21:58:26 oak-gw06 kernel: mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v2.2-1 (Feb 2014) May 1 21:58:26 oak-gw06 kernel: check_flow_steering_support: Device managed flow steering is unavailable for IB port in multifunction env. May 1 21:58:26 oak-gw06 kernel: mlx4_ib_add: counter index 59 for port 1 allocated 0 May 1 21:58:26 oak-gw06 kernel: mlx4_core 0000:00:07.0: mlx4_ib: multi-function enabled May 1 21:58:26 oak-gw06 kernel: mlx4_core 0000:00:07.0: mlx4_ib: operating in qp1 tunnel mode May 1 21:58:27 oak-gw06 systemd: Started Initialize the iWARP/InfiniBand/RDMA stack in the kernel. May 1 21:58:28 oak-gw06 kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team May 1 21:58:28 oak-gw06 kernel: Ebtables v2.0 registered May 1 21:58:28 oak-gw06 kernel: nf_conntrack version 0.5.0 (65536 buckets, 262144 max) May 1 21:58:28 oak-gw06 kernel: bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this. May 1 21:58:28 oak-gw06 kernel: Netfilter messages via NETLINK v0.30. May 1 21:58:28 oak-gw06 kernel: ip_set: protocol 6 May 1 21:58:29 oak-gw06 kernel: bnx2x 0000:00:06.0 ens6: using MSI-X IRQs: fp[0] 36 ... fp[1] 37 May 1 21:58:29 oak-gw06 kernel: bnx2x 0000:00:06.0 ens6: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit May 1 21:58:39 oak-gw06 kernel: IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready May 1 21:58:39 oak-gw06 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready May 1 21:58:44 oak-gw06 kernel: FS-Cache: Loaded May 1 21:58:44 oak-gw06 kernel: FS-Cache: Netfs 'nfs' registered for caching May 1 21:58:44 oak-gw06 kernel: Key type dns_resolver registered May 1 21:58:44 oak-gw06 kernel: NFS: Registering the id_resolver key type May 1 21:58:44 oak-gw06 kernel: Key type id_resolver registered May 1 21:58:44 oak-gw06 kernel: Key type id_legacy registered May 1 21:58:44 oak-gw06 systemd: Starting Crash recovery kernel arming... May 1 21:58:46 oak-gw06 kernel: libcfs: loading out-of-tree module taints kernel. May 1 21:58:46 oak-gw06 kernel: libcfs: module verification failed: signature and/or required key missing - tainting kernel May 1 21:58:46 oak-gw06 kernel: LNet: HW CPU cores: 8, npartitions: 1 May 1 21:58:46 oak-gw06 kernel: alg: No test for adler32 (adler32-zlib) May 1 21:58:46 oak-gw06 kernel: alg: No test for crc32 (crc32-table) May 1 21:58:49 oak-gw06 kdumpctl: kexec: loaded kdump kernel May 1 21:58:49 oak-gw06 systemd: Started Crash recovery kernel arming. May 1 21:58:51 oak-gw06 kernel: sha512_ssse3: Using AVX2 optimized SHA-512 implementation May 1 21:58:54 oak-gw06 kernel: Lustre: Lustre: Build Version: 2.9.0_srcc6 May 1 21:58:54 oak-gw06 kernel: LNet: Added LNI 10.0.2.225@o2ib5 [8/256/0/180] May 1 21:59:21 oak-gw06 kernel: Lustre: Mounted oak-client May 1 21:59:21 oak-gw06 systemd: Startup finished in 1.102s (kernel) + 1.483s (initrd) + 59.406s (userspace) = 1min 1.992s. May 1 22:37:31 oak-gw06 kernel: conntrack: generic helper won't handle protocol 47. Please consider loading the specific helper module. May 7 17:04:02 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494201835/real 1494201835] req@ffff880399574300 x1566259457522656/t0(0) o400->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1494201842 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 7 17:04:02 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494201835/real 1494201835] req@ffff880399576100 x1566259457522592/t0(0) o400->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1494201842 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 7 17:04:02 oak-gw06 kernel: Lustre: 1769:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494201835/real 1494201835] req@ffff880399577c00 x1566259457522752/t0(0) o400->oak-OST000a-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1494201842 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 7 17:04:02 oak-gw06 kernel: Lustre: 1769:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 1 previous similar message May 7 17:04:02 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 1 previous similar message May 7 17:04:02 oak-gw06 kernel: Lustre: oak-OST0006-osc-ffff88041b99c000: Connection to oak-OST0006 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete May 7 17:04:02 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 6 previous similar messages May 7 17:04:08 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494201842/real 0] req@ffff880413f7e700 x1566259457523120/t0(0) o8->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494201848 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 May 7 17:04:08 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 2 previous similar messages May 7 17:04:46 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3126:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds May 7 17:04:46 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3189:kiblnd_check_conns()) Timed out RDMA with 10.0.2.101@o2ib5 (51): c: 0, oc: 0, rc: 8 May 7 17:05:28 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494201917/real 0] req@ffff8803b8340000 x1566259457523712/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494201928 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 May 7 17:05:28 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 11 previous similar messages May 7 17:06:57 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494202017/real 1494202017] req@ffff880274c7c000 x1566259457525056/t0(0) o8->oak-OST000c-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494202028 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 7 17:06:57 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 3 previous similar messages May 7 17:07:47 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494202067/real 1494202067] req@ffff88030189ea00 x1566259457525696/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494202088 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 7 17:07:47 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 11 previous similar messages May 7 17:08:37 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494202117/real 1494202117] req@ffff8803f62cf900 x1566259457526752/t0(0) o8->oak-OST000e-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494202138 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 7 17:08:37 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 11 previous similar messages May 7 17:09:27 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494202167/real 1494202167] req@ffff8802296c5200 x1566259457527360/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494202198 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 7 17:09:27 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 11 previous similar messages May 7 17:10:17 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494202217/real 1494202217] req@ffff88028c719e00 x1566259457528256/t0(0) o8->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494202248 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 7 17:10:17 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 11 previous similar messages May 7 17:11:57 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494202317/real 1494202317] req@ffff8803ae3c5500 x1566259457529856/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494202363 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 7 17:11:57 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 23 previous similar messages May 7 17:14:27 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494202467/real 1494202467] req@ffff8803d48d9500 x1566259457532352/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494202522 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 7 17:14:27 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 35 previous similar messages May 7 17:19:27 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494202767/real 1494202767] req@ffff880285a19e00 x1566259457537408/t0(0) o8->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494202823 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 7 17:19:27 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 71 previous similar messages May 7 17:28:37 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494203317/real 1494203317] req@ffff88040408d500 x1566259457546496/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494203363 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 7 17:28:37 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 131 previous similar messages May 7 17:36:49 oak-gw06 kernel: Lustre: oak-OST0012-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) May 7 17:36:51 oak-gw06 kernel: Lustre: oak-OST0016-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) May 7 17:36:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message May 7 17:36:54 oak-gw06 kernel: Lustre: oak-OST000a-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) May 7 17:36:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message May 7 17:36:57 oak-gw06 kernel: Lustre: oak-OST0008-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) May 7 17:36:57 oak-gw06 kernel: Lustre: Skipped 3 previous similar messages May 11 10:11:42 oak-gw06 kernel: Lustre: 1767:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494522695/real 1494522695] req@ffff8803f62ccc00 x1566259780669056/t0(0) o400->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1494522702 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 11 10:11:42 oak-gw06 kernel: Lustre: 1768:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494522695/real 1494522695] req@ffff8803c9a56d00 x1566259780669344/t0(0) o400->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1494522702 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 11 10:11:42 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494522695/real 1494522695] req@ffff8803f62cdb00 x1566259780669024/t0(0) o400->oak-OST0002-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1494522702 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 11 10:11:42 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 95 previous similar messages May 11 10:11:42 oak-gw06 kernel: Lustre: 1768:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 95 previous similar messages May 11 10:11:42 oak-gw06 kernel: Lustre: oak-OST0002-osc-ffff88041b99c000: Connection to oak-OST0002 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete May 11 10:11:42 oak-gw06 kernel: Lustre: Skipped 12 previous similar messages May 11 10:11:42 oak-gw06 kernel: Lustre: 1767:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 9 previous similar messages May 11 10:12:32 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494522752/real 1494522752] req@ffff8803b2b82400 x1566259780670256/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494522763 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 10:12:32 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 18 previous similar messages May 11 10:13:17 oak-gw06 kernel: Lustre: oak-OST0018-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) May 11 10:13:17 oak-gw06 kernel: Lustre: Skipped 3 previous similar messages May 11 10:13:22 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494522802/real 1494522802] req@ffff8803b2b82700 x1566259780671520/t0(0) o8->oak-OST000c-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494522818 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 10:13:22 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 14 previous similar messages May 11 10:13:45 oak-gw06 kernel: Lustre: oak-OST001a-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) May 11 10:14:12 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494522852/real 1494522852] req@ffff8803b2b81200 x1566259780672464/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494522873 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 10:14:12 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 13 previous similar messages May 11 10:14:13 oak-gw06 kernel: Lustre: oak-OST0008-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) May 11 10:14:37 oak-gw06 kernel: Lustre: oak-OST0006-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) May 11 10:14:37 oak-gw06 kernel: Lustre: Skipped 1 previous similar message May 11 10:15:02 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494522902/real 1494522902] req@ffff8801c5237300 x1566259780673600/t0(0) o8->oak-OST0002-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494522928 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 10:15:02 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 10 previous similar messages May 11 10:15:24 oak-gw06 kernel: Lustre: oak-OST000c-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) May 11 10:15:24 oak-gw06 kernel: Lustre: Skipped 1 previous similar message May 11 10:15:49 oak-gw06 kernel: Lustre: oak-OST0010-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) May 11 10:15:52 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494522952/real 1494522952] req@ffff8800afd26d00 x1566259780675120/t0(0) o8->oak-OST001c-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494522983 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 10:15:52 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 7 previous similar messages May 11 10:16:07 oak-gw06 kernel: Lustre: oak-OST0012-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) May 11 10:16:41 oak-gw06 kernel: Lustre: oak-OST000e-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) May 11 10:16:42 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494523002/real 1494523002] req@ffff880301a5b300 x1566259780675984/t0(0) o8->oak-OST000a-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494523038 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 10:16:42 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 5 previous similar messages May 11 10:18:00 oak-gw06 kernel: Lustre: oak-OST000a-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) May 11 10:18:00 oak-gw06 kernel: Lustre: Skipped 4 previous similar messages May 11 12:22:31 oak-gw06 kernel: Lustre: 1768:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494530527/real 1494530551] req@ffff8803cef38000 x1566259780830048/t0(0) o400->oak-OST0006-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1494530728 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 12:22:31 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494530527/real 1494530551] req@ffff8803cef3b600 x1566259780830064/t0(0) o400->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1494530728 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 12:22:31 oak-gw06 kernel: Lustre: oak-OST0007-osc-ffff88041b99c000: Connection to oak-OST0007 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete May 11 12:22:31 oak-gw06 kernel: Lustre: Skipped 13 previous similar messages May 11 12:22:31 oak-gw06 kernel: Lustre: 1768:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 45 previous similar messages May 11 12:22:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494530576/real 1494530576] req@ffff880408c42100 x1566259780830944/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494530582 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 12:22:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 13 previous similar messages May 11 12:23:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494530601/real 1494530601] req@ffff8800ba1cde00 x1566259780831456/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494530612 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 12:23:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 29 previous similar messages May 11 12:23:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494530626/real 1494530626] req@ffff88009febe100 x1566259780831968/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494530637 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 12:23:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 29 previous similar messages May 11 12:24:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494530651/real 1494530651] req@ffff880393f65200 x1566259780832480/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494530667 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 12:24:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 29 previous similar messages May 11 12:24:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494530676/real 1494530676] req@ffff880411bbb300 x1566259780832992/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494530692 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 12:24:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 29 previous similar messages May 11 12:25:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494530726/real 1494530726] req@ffff880277292700 x1566259780834016/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494530747 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 12:25:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 59 previous similar messages May 11 12:26:41 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494530801/real 1494530801] req@ffff8801edb1db00 x1566259780835552/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494530832 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 12:26:41 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 89 previous similar messages May 11 12:29:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494530951/real 1494530951] req@ffff880252be7c00 x1566259780838624/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494530997 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 12:29:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 179 previous similar messages May 11 12:32:58 oak-gw06 kernel: Lustre: oak-OST001c-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) May 11 12:33:38 oak-gw06 kernel: Lustre: oak-OST0002-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) May 11 12:33:38 oak-gw06 kernel: Lustre: Skipped 11 previous similar messages May 11 12:34:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494531251/real 1494531251] req@ffff8800afd25b00 x1566259780843840/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494531307 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 11 12:34:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 211 previous similar messages May 11 12:35:56 oak-gw06 kernel: Lustre: oak-OST0000-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) May 11 12:35:56 oak-gw06 kernel: Lustre: Skipped 1 previous similar message May 11 12:37:11 oak-gw06 kernel: Lustre: oak-OST0003-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) May 12 13:39:11 oak-gw06 kernel: LustreError: 11-0: oak-OST000f-osc-ffff88041b99c000: operation obd_ping to node 10.0.2.101@o2ib5 failed: rc = -107 May 12 13:39:11 oak-gw06 kernel: Lustre: oak-OST0005-osc-ffff88041b99c000: Connection to oak-OST0005 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete May 12 13:39:11 oak-gw06 kernel: Lustre: Skipped 29 previous similar messages May 12 13:39:11 oak-gw06 kernel: LustreError: Skipped 12 previous similar messages May 12 13:39:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1494621576/real 1494621576] req@ffff88028f214000 x1566259785480160/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1494621582 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 May 12 13:39:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 46 previous similar messages May 12 13:40:17 oak-gw06 kernel: Lustre: oak-OST000d-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) May 12 13:40:17 oak-gw06 kernel: Lustre: Skipped 14 previous similar messages May 12 13:40:35 oak-gw06 kernel: Lustre: oak-OST0007-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) May 12 13:40:35 oak-gw06 kernel: Lustre: Skipped 9 previous similar messages May 12 13:41:32 oak-gw06 kernel: Lustre: oak-OST0009-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) May 12 13:41:32 oak-gw06 kernel: Lustre: Skipped 3 previous similar messages May 12 16:25:09 oak-gw06 kernel: Lustre: DEBUG MARKER: Fri May 12 16:25:09 2017 May 19 15:12:59 oak-gw06 kernel: Peer 128.218.42.180:58982/50918 unexpectedly shrunk window 664054312:664054318 (repaired) May 25 09:34:26 oak-gw06 kernel: LNetError: 20214:0:(o2iblnd_cb.c:2749:kiblnd_rejected()) 10.0.2.204@o2ib5 rejected: o2iblnd fatal error Jun 6 08:19:43 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1496762358/real 1496762383] req@ffff880393bc5800 x1566264108474208/t0(0) o400->oak-OST0003-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1496763114 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 6 08:19:43 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1496762358/real 1496762383] req@ffff880393bc4c00 x1566264108474240/t0(0) o400->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1496763114 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 6 08:19:43 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jun 6 08:19:43 oak-gw06 kernel: Lustre: oak-OST001b-osc-ffff88041b99c000: Connection to oak-OST001b (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 6 08:19:43 oak-gw06 kernel: Lustre: Skipped 15 previous similar messages Jun 6 08:19:43 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 28 previous similar messages Jun 6 08:20:33 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1496762433/real 1496762433] req@ffff8801f7678c00 x1566264108475680/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1496762444 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 6 08:20:33 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jun 6 08:21:23 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1496762483/real 1496762483] req@ffff8801670c8900 x1566264108476896/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1496762499 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 6 08:21:23 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 6 08:22:13 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1496762533/real 1496762533] req@ffff88016c82e700 x1566264108478112/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1496762554 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 6 08:22:13 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 6 08:23:03 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1496762583/real 1496762583] req@ffff88022b7bc900 x1566264108479328/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1496762609 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 6 08:23:03 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 6 08:23:53 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1496762633/real 1496762633] req@ffff8803d3a9cf00 x1566264108480544/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1496762664 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 6 08:23:53 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 6 08:24:43 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1496762683/real 1496762683] req@ffff88033ae70300 x1566264108481760/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1496762719 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 6 08:24:43 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 6 08:26:23 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1496762783/real 1496762783] req@ffff880393bc4f00 x1566264108484192/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1496762829 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 6 08:26:23 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 35 previous similar messages Jun 6 08:28:53 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1496762933/real 1496762933] req@ffff88039b91f600 x1566264108487840/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1496762988 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 6 08:28:53 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 53 previous similar messages Jun 6 08:33:48 oak-gw06 kernel: Lustre: oak-OST001b-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 6 08:34:09 oak-gw06 kernel: Lustre: oak-OST0005-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 6 08:34:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jun 6 08:34:33 oak-gw06 kernel: Lustre: oak-OST000d-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 6 08:34:33 oak-gw06 kernel: Lustre: Skipped 14 previous similar messages Jun 13 01:14:56 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1497341689/real 0] req@ffff880270a7db00 x1566264123271856/t0(0) o400->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1497341696 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 13 01:14:56 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1497341689/real 0] req@ffff8802cbf99b00 x1566264123271376/t0(0) o400->oak-OST0003-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1497341696 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 13 01:14:56 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 90 previous similar messages Jun 13 01:14:56 oak-gw06 kernel: Lustre: oak-OST0011-osc-ffff88041b99c000: Connection to oak-OST0011 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 13 01:14:56 oak-gw06 kernel: Lustre: Skipped 15 previous similar messages Jun 13 01:14:56 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jun 13 01:15:02 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1497341696/real 0] req@ffff880077662100 x1566264123272160/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497341702 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 13 01:15:02 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 13 01:15:49 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3126:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 10 seconds Jun 13 01:15:49 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3189:kiblnd_check_conns()) Timed out RDMA with 10.0.2.102@o2ib5 (60): c: 0, oc: 0, rc: 8 Jun 13 01:16:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497341796/real 1497341796] req@ffff88030a6c4c00 x1566264123273488/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497341807 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 01:16:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 13 01:17:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497341846/real 1497341846] req@ffff880363d33000 x1566264123274704/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497341862 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 01:17:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 13 01:18:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497341896/real 1497341896] req@ffff880062d9cc00 x1566264123275920/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497341917 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 01:18:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 13 01:19:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497341996/real 1497341996] req@ffff8801065b9500 x1566264123278352/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497342027 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 01:19:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 35 previous similar messages Jun 13 01:22:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497342146/real 1497342146] req@ffff8800afd27300 x1566264123282000/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497342192 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 01:22:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 53 previous similar messages Jun 13 01:27:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497342446/real 1497342446] req@ffff8801d17ed200 x1566264123289424/t0(0) o8->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497342502 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 01:27:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 107 previous similar messages Jun 13 01:37:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497343046/real 1497343046] req@ffff8801ddc2db00 x1566264123303888/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497343077 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 01:37:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 215 previous similar messages Jun 13 01:47:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497343646/real 1497343646] req@ffff8802e5ac0600 x1566264123318736/t0(0) o8->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497343702 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 01:47:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 01:57:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497344246/real 1497344246] req@ffff88030189e400 x1566264123333584/t0(0) o8->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497344292 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 01:57:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 02:08:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497344896/real 1497344896] req@ffff8801d3637900 x1566264123348880/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497344952 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 02:08:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 217 previous similar messages Jun 13 02:18:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497345496/real 1497345496] req@ffff8802903e1500 x1566264123363728/t0(0) o8->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497345551 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 02:18:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 02:28:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497346096/real 1497346096] req@ffff88041bcd1200 x1566264123378320/t0(0) o8->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497346117 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 02:28:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 215 previous similar messages Jun 13 02:39:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497346746/real 1497346746] req@ffff880237e42400 x1566264123393872/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497346802 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 02:39:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 225 previous similar messages Jun 13 02:49:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497347346/real 1497347346] req@ffff8801bb8d0600 x1566264123408464/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497347387 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 02:49:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 215 previous similar messages Jun 13 02:59:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497347946/real 1497347946] req@ffff88021f764900 x1566264123423312/t0(0) o8->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497348002 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 02:59:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 03:09:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497348546/real 1497348546] req@ffff8801b2a24600 x1566264123438160/t0(0) o8->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497348601 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 03:09:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 03:19:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497349196/real 1497349196] req@ffff8803f57af000 x1566264123453456/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497349212 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 03:19:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 217 previous similar messages Jun 13 03:29:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497349796/real 1497349796] req@ffff88024ab00c00 x1566264123468304/t0(0) o8->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497349852 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 03:29:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 03:39:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497350396/real 1497350396] req@ffff880183b02a00 x1566264123483152/t0(0) o8->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497350427 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 03:39:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 03:49:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497350996/real 1497350996] req@ffff8802d1a67300 x1566264123497776/t0(0) o8->oak-OST0023-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497351052 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 03:49:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 215 previous similar messages Jun 13 03:59:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497351596/real 1497351596] req@ffff880400b30000 x1566264123512336/t0(0) o8->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497351642 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 03:59:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 215 previous similar messages Jun 13 04:10:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497352246/real 1497352246] req@ffff880366fdcf00 x1566264123527632/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497352302 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 04:10:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 217 previous similar messages Jun 13 04:20:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497352846/real 1497352846] req@ffff880062d9db00 x1566264123542480/t0(0) o8->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497352901 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 04:20:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 04:30:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497353446/real 1497353446] req@ffff8800b9adc000 x1566264123557104/t0(0) o8->oak-OST0013-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497353467 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 04:30:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 215 previous similar messages Jun 13 04:41:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497354096/real 1497354096] req@ffff8801d17ef900 x1566264123572624/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497354152 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 04:41:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 225 previous similar messages Jun 13 04:51:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497354696/real 1497354696] req@ffff8802d1a64300 x1566264123587472/t0(0) o8->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497354737 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 04:51:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 05:01:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497355296/real 1497355296] req@ffff88009b75a700 x1566264123602064/t0(0) o8->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497355352 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 05:01:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 215 previous similar messages Jun 13 05:11:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497355896/real 1497355896] req@ffff8803f57aea00 x1566264123616912/t0(0) o8->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497355951 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 05:11:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 05:21:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497356496/real 1497356496] req@ffff88030189f000 x1566264123631536/t0(0) o8->oak-OST0023-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497356507 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 05:21:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 215 previous similar messages Jun 13 05:31:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497357096/real 1497357096] req@ffff88008a77aa00 x1566264123646096/t0(0) o8->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497357152 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 05:31:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 215 previous similar messages Jun 13 05:42:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497357746/real 1497357746] req@ffff8801d17ec300 x1566264123661392/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497357777 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 05:42:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 217 previous similar messages Jun 13 05:52:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497358346/real 1497358346] req@ffff8801d17edb00 x1566264123676240/t0(0) o8->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497358402 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 05:52:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 06:02:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497358946/real 1497358946] req@ffff88041d082a00 x1566264123691088/t0(0) o8->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497358992 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 06:02:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 06:13:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497359596/real 1497359596] req@ffff8802ebdba400 x1566264123706384/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497359652 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 06:13:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 217 previous similar messages Jun 13 06:23:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497360196/real 1497360196] req@ffff880306644900 x1566264123721232/t0(0) o8->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497360251 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 06:23:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 06:33:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497360796/real 1497360796] req@ffff88005c102700 x1566264123735824/t0(0) o8->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497360817 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 06:33:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 215 previous similar messages Jun 13 06:43:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497361396/real 1497361396] req@ffff88020efe5200 x1566264123750672/t0(0) o8->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497361452 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 06:43:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 06:54:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497362046/real 1497362046] req@ffff8803bb1d3900 x1566264123765968/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497362087 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 06:54:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 217 previous similar messages Jun 13 07:04:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497362646/real 1497362646] req@ffff880062d9e400 x1566264123781040/t0(0) o8->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497362702 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 07:04:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 07:14:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497363246/real 1497363246] req@ffff880382fb4000 x1566264123795408/t0(0) o8->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497363301 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 07:14:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 215 previous similar messages Jun 13 07:24:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497363846/real 1497363846] req@ffff8803f62ce100 x1566264123810256/t0(0) o8->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497363857 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 07:24:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 07:34:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497364496/real 1497364496] req@ffff880347294000 x1566264123825552/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497364552 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 07:34:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 217 previous similar messages Jun 13 07:44:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497365096/real 1497365096] req@ffff8802f0539200 x1566264123840400/t0(0) o8->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497365127 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 07:44:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 07:55:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497365746/real 1497365746] req@ffff88000efa7c00 x1566264123855952/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497365802 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 07:55:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 225 previous similar messages Jun 13 08:05:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497366346/real 1497366346] req@ffff880347295e00 x1566264123870992/t0(0) o8->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497366397 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 13 08:05:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 223 previous similar messages Jun 13 08:16:42 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497366946/real 1497366946] req@ffff880062d9d800 x1566264123885136/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497367002 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 13 08:16:42 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 207 previous similar messages Jun 13 08:19:33 oak-gw06 kernel: Lustre: oak-OST0009-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 13 08:19:37 oak-gw06 kernel: Lustre: oak-OST001d-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 13 08:19:37 oak-gw06 kernel: Lustre: Skipped 2 previous similar messages Jun 13 08:19:48 oak-gw06 kernel: Lustre: oak-OST0007-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 13 08:19:48 oak-gw06 kernel: Lustre: Skipped 8 previous similar messages Jun 14 17:04:13 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497485046/real 1497485046] req@ffff880400f6aa00 x1566264126755744/t0(0) o400->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1497485053 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 14 17:04:13 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497485046/real 1497485046] req@ffff8803daff5500 x1566264126756096/t0(0) o400->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1497485053 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 14 17:04:13 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 14 17:04:13 oak-gw06 kernel: Lustre: oak-OST0001-osc-ffff88041b99c000: Connection to oak-OST0001 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 14 17:04:13 oak-gw06 kernel: Lustre: Skipped 19 previous similar messages Jun 14 17:04:13 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jun 14 17:05:06 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3126:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 10 seconds Jun 14 17:05:06 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3189:kiblnd_check_conns()) Timed out RDMA with 10.0.2.102@o2ib5 (60): c: 0, oc: 0, rc: 8 Jun 14 17:07:33 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497485253/real 1497485253] req@ffff880379723c00 x1566264126759104/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497485264 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 14 17:07:33 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jun 14 17:08:23 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497485303/real 1497485303] req@ffff8803f62ce400 x1566264126760320/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497485319 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 14 17:08:23 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 14 17:09:13 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497485353/real 1497485353] req@ffff88007eaf4f00 x1566264126761536/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497485374 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 14 17:09:13 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 14 17:10:53 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497485453/real 1497485453] req@ffff880400f69e00 x1566264126763968/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497485484 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 14 17:10:53 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 35 previous similar messages Jun 14 17:13:23 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497485603/real 1497485603] req@ffff8802b770cc00 x1566264126767616/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497485649 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 14 17:13:23 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 53 previous similar messages Jun 14 17:18:23 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497485903/real 1497485903] req@ffff8803ec2c4000 x1566264126775200/t0(0) o8->oak-OST0013-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497485959 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 14 17:18:23 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 115 previous similar messages Jun 14 17:21:43 oak-gw06 kernel: Lustre: oak-OST001f-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 14 17:21:43 oak-gw06 kernel: Lustre: Skipped 5 previous similar messages Jun 14 17:21:46 oak-gw06 kernel: Lustre: oak-OST0023-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 14 17:21:46 oak-gw06 kernel: Lustre: Skipped 11 previous similar messages Jun 14 17:21:51 oak-gw06 kernel: Lustre: oak-OST0011-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 14 17:23:04 oak-gw06 kernel: Lustre: oak-OST0021-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 14 17:23:04 oak-gw06 kernel: Lustre: Skipped 2 previous similar messages Jun 18 05:42:45 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497789009/real 1497789009] req@ffff8803c6328c00 x1566264134146400/t0(0) o400->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1497789765 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 05:42:45 oak-gw06 kernel: Lustre: 1767:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497789009/real 1497789009] req@ffff8803c6328000 x1566264134146336/t0(0) o400->oak-OST0013-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1497789765 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 05:42:45 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497789009/real 1497789009] req@ffff88031b0ccc00 x1566264134146048/t0(0) o400->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1497789765 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 05:42:45 oak-gw06 kernel: Lustre: oak-OST0005-osc-ffff88041b99c000: Connection to oak-OST0005 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 18 05:42:45 oak-gw06 kernel: Lustre: 1767:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 48 previous similar messages Jun 18 05:42:45 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 48 previous similar messages Jun 18 05:42:45 oak-gw06 kernel: Lustre: Skipped 16 previous similar messages Jun 18 05:42:45 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jun 18 05:42:51 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497789765/real 1497789765] req@ffff8801e0fe4300 x1566264134165104/t0(0) o8->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497789771 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 05:42:51 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 18 05:43:10 oak-gw06 kernel: Lustre: 1768:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497789034/real 1497789034] req@ffff880083178000 x1566264134147136/t0(0) o400->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1497789790 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 05:43:10 oak-gw06 kernel: Lustre: 1768:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 26 previous similar messages Jun 18 05:43:35 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497789059/real 1497789059] req@ffff88008317bc00 x1566264134147456/t0(0) o400->oak-OST000d-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1497789815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 05:43:35 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jun 18 05:43:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497789815/real 1497789815] req@ffff88034fba1b00 x1566264134165920/t0(0) o8->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497789826 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 05:43:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jun 18 05:44:25 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497789109/real 1497789109] req@ffff8800a4b9c300 x1566264134148736/t0(0) o400->oak-OST0011-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1497789865 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 05:44:25 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 48 previous similar messages Jun 18 05:45:15 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497789159/real 1497789159] req@ffff88006d7ae100 x1566264134149728/t0(0) o400->oak-OST0003-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1497789915 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 05:45:15 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 49 previous similar messages Jun 18 05:46:31 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497789234/real 1497789234] req@ffff880083179500 x1566264134151552/t0(0) o400->oak-OST0003-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1497789990 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 05:46:31 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 75 previous similar messages Jun 18 05:49:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497790115/real 1497790115] req@ffff880232fd2100 x1566264134172704/t0(0) o8->oak-OST000d-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497790151 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 05:49:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 147 previous similar messages Jun 18 05:54:25 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497789709/real 1497789709] req@ffff88016c8dd500 x1566264134163104/t0(0) o400->oak-OST0003-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1497790465 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 05:54:25 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 293 previous similar messages Jun 18 06:05:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497791090/real 1497791090] req@ffff88030189e700 x1566264134191328/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497791106 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 06:05:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 163 previous similar messages Jun 18 06:15:20 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497791665/real 1497791665] req@ffff8800b9a59e00 x1566264134203296/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497791720 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 06:15:20 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 06:25:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497792315/real 1497792315] req@ffff8800b9a58c00 x1566264134215904/t0(0) o8->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497792331 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 06:25:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 06:35:45 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497792890/real 1497792890] req@ffff88025d20d200 x1566264134228000/t0(0) o8->oak-OST0019-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497792945 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 06:35:45 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 06:45:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497793540/real 1497793540] req@ffff8803bfd26400 x1566264134240448/t0(0) o8->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497793556 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 06:45:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 06:56:10 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497794115/real 1497794115] req@ffff88026b6db900 x1566264134251936/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497794170 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 06:56:10 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 07:06:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497794765/real 1497794765] req@ffff8803e733f900 x1566264134264288/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497794781 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 07:06:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 07:16:35 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497795340/real 1497795340] req@ffff8801b8291b00 x1566264134276256/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497795395 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 07:16:35 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 07:26:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497795990/real 1497795990] req@ffff8800bacbb900 x1566264134289184/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497796016 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 07:26:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 07:37:00 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497796565/real 1497796565] req@ffff8803bd426400 x1566264134300864/t0(0) o8->oak-OST0013-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497796620 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 07:37:00 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 07:47:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497797215/real 1497797215] req@ffff880062d9d200 x1566264134314016/t0(0) o8->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497797241 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 07:47:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 07:57:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497797790/real 1497797790] req@ffff88034f9f1200 x1566264134325216/t0(0) o8->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497797846 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 07:57:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 08:07:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497798440/real 1497798440] req@ffff88039a77d200 x1566264134338208/t0(0) o8->oak-OST0019-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497798466 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 08:07:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 08:17:51 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497799015/real 1497799015] req@ffff8803f62cf000 x1566264134349440/t0(0) o8->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497799071 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 08:17:51 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 08:28:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497799665/real 1497799665] req@ffff88022ae03600 x1566264134362368/t0(0) o8->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497799691 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 08:28:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 08:38:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497800240/real 1497800240] req@ffff8800b73ccf00 x1566264134373536/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497800296 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 08:38:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 08:49:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497800940/real 1497800940] req@ffff8800b9a58000 x1566264134387680/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497800976 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 08:49:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 161 previous similar messages Jun 18 09:00:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497801565/real 1497801565] req@ffff88025cec7000 x1566264134400256/t0(0) o8->oak-OST0023-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497801621 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 09:00:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 09:11:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497802240/real 1497802240] req@ffff88041fd3f000 x1566264134414080/t0(0) o8->oak-OST0023-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497802281 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 09:11:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 161 previous similar messages Jun 18 09:22:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497802890/real 1497802890] req@ffff8802b8b43c00 x1566264134425888/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497802946 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 09:22:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 09:33:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497803540/real 1497803540] req@ffff88028dbcf300 x1566264134439392/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497803586 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 09:33:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 161 previous similar messages Jun 18 09:44:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497804215/real 1497804215] req@ffff8803b673c300 x1566264134452064/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497804271 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 09:44:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 09:54:51 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497804840/real 1497804840] req@ffff880270ad7c00 x1566264134465248/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497804891 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 09:54:51 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 161 previous similar messages Jun 18 10:05:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497805490/real 1497805490] req@ffff8801968ea400 x1566264134478080/t0(0) o8->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497805506 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 10:05:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 10:15:20 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497806065/real 1497806065] req@ffff8802dc8e1b00 x1566264134489568/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497806120 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 10:15:20 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 10:25:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497806715/real 1497806715] req@ffff8802660cd200 x1566264134501920/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497806731 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 10:25:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 10:35:45 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497807290/real 1497807290] req@ffff880366ffa700 x1566264134513888/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497807345 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 10:35:45 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 10:45:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497807940/real 1497807940] req@ffff8802b770cf00 x1566264134526592/t0(0) o8->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497807956 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 10:45:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 10:56:10 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497808515/real 1497808515] req@ffff880386614000 x1566264134538496/t0(0) o8->oak-OST0013-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497808570 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 10:56:10 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 11:06:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497809165/real 1497809165] req@ffff8801c2ea7300 x1566264134550560/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497809181 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 11:06:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 11:16:35 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497809740/real 1497809740] req@ffff880128d27c00 x1566264134562528/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497809795 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 11:16:35 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 11:26:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497810390/real 1497810390] req@ffff88022ae00600 x1566264134575776/t0(0) o8->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497810416 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 11:26:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 11:37:00 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497810965/real 1497810965] req@ffff880252bbc000 x1566264134587136/t0(0) o8->oak-OST0013-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497811020 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 11:37:00 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 11:47:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497811615/real 1497811615] req@ffff8801daf0fc00 x1566264134599776/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497811641 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 11:47:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 11:57:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497812190/real 1497812190] req@ffff88023c2fe400 x1566264134611648/t0(0) o8->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497812246 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 11:57:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 12:07:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497812840/real 1497812840] req@ffff8802660cf000 x1566264134624320/t0(0) o8->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497812866 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 12:07:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 12:17:51 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497813415/real 1497813415] req@ffff8803f5bdc000 x1566264134635776/t0(0) o8->oak-OST0013-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497813471 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 12:17:51 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 12:28:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497814065/real 1497814065] req@ffff880420f04000 x1566264134648704/t0(0) o8->oak-OST0013-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497814091 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 12:28:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Jun 18 12:38:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497814640/real 1497814640] req@ffff8802660ccc00 x1566264134660160/t0(0) o8->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497814696 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 12:38:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 12:49:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497815340/real 1497815340] req@ffff88027ba5ea00 x1566264134674016/t0(0) o8->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497815376 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 12:49:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 161 previous similar messages Jun 18 13:00:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497815965/real 1497815965] req@ffff880181f39e00 x1566264134685984/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497816021 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 13:00:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 13:11:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497816640/real 1497816640] req@ffff880374ec1800 x1566264134700192/t0(0) o8->oak-OST0019-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497816681 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 13:11:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 161 previous similar messages Jun 18 13:22:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497817290/real 1497817290] req@ffff8801c52acf00 x1566264134712160/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497817346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 13:22:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 13:33:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497817940/real 1497817940] req@ffff88028aaf9200 x1566264134725664/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497817986 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 13:33:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 161 previous similar messages Jun 18 13:44:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497818615/real 1497818615] req@ffff88004f2aed00 x1566264134738656/t0(0) o8->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497818671 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 13:44:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 13:54:51 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497819240/real 1497819240] req@ffff880399574f00 x1566264134751520/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497819291 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 13:54:51 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 161 previous similar messages Jun 18 14:05:40 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497819940/real 1497819940] req@ffff8803f62cf300 x1566264134768064/t0(0) o8->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497819961 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 18 14:05:40 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 233 previous similar messages Jun 18 14:16:30 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497820590/real 1497820590] req@ffff8802e5b87600 x1566264134783776/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497820646 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 18 14:16:30 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 233 previous similar messages Jun 18 14:29:47 oak-gw06 kernel: Lustre: oak-OST000b-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 18 14:29:47 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jun 18 14:29:49 oak-gw06 kernel: Lustre: oak-OST0013-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 18 14:29:49 oak-gw06 kernel: Lustre: Skipped 2 previous similar messages Jun 18 14:29:54 oak-gw06 kernel: Lustre: oak-OST0009-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 18 14:29:54 oak-gw06 kernel: Lustre: Skipped 3 previous similar messages Jun 18 14:30:02 oak-gw06 kernel: Lustre: oak-OST000d-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 18 14:30:02 oak-gw06 kernel: Lustre: Skipped 8 previous similar messages Jun 18 17:43:10 oak-gw06 kernel: LustreError: 11-0: oak-MDT0000-mdc-ffff88041b99c000: operation obd_ping to node 10.0.2.52@o2ib5 failed: rc = -107 Jun 18 17:43:10 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Jun 18 17:43:10 oak-gw06 kernel: Lustre: oak-MDT0000-mdc-ffff88041b99c000: Connection to oak-MDT0000 (at 10.0.2.52@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 18 17:43:10 oak-gw06 kernel: Lustre: Skipped 15 previous similar messages Jun 18 17:44:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497833040/real 1497833040] req@ffff88010a8d9200 x1566264135082032/t0(0) o38->oak-MDT0000-mdc-ffff88041b99c000@10.0.2.51@o2ib5:12/10 lens 520/544 e 0 to 1 dl 1497833046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 17:44:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Jun 18 17:45:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497833140/real 1497833140] req@ffff88018159b600 x1566264135084464/t0(0) o38->oak-MDT0000-mdc-ffff88041b99c000@10.0.2.51@o2ib5:12/10 lens 520/544 e 0 to 1 dl 1497833156 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 17:45:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 18 17:49:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497833315/real 1497833315] req@ffff8801d3722700 x1566264135088704/t0(0) o38->oak-MDT0000-mdc-ffff88041b99c000@10.0.2.51@o2ib5:12/10 lens 520/544 e 0 to 1 dl 1497833346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 18 17:49:06 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jun 18 17:54:00 oak-gw06 kernel: Lustre: oak-MDT0000-mdc-ffff88041b99c000: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jun 18 17:54:00 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jun 19 11:47:24 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1497898037/real 0] req@ffff880396b80000 x1566264203572432/t0(0) o103->oak-OST0020-osc-ffff88041b99c000@10.0.2.101@o2ib5:17/18 lens 408/224 e 0 to 1 dl 1497898044 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 19 11:47:24 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1497898037/real 0] req@ffff8800add14c00 x1566264203572448/t0(0) o103->oak-OST001e-osc-ffff88041b99c000@10.0.2.101@o2ib5:17/18 lens 424/224 e 0 to 1 dl 1497898044 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 19 11:47:24 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1497898037/real 0] req@ffff88041bfc4f00 x1566264203572480/t0(0) o103->oak-OST000c-osc-ffff88041b99c000@10.0.2.101@o2ib5:17/18 lens 448/224 e 0 to 1 dl 1497898044 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 19 11:47:24 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1497898037/real 0] req@ffff88039bba2400 x1566264203572944/t0(0) o103->oak-OST0002-osc-ffff88041b99c000@10.0.2.101@o2ib5:17/18 lens 464/224 e 0 to 1 dl 1497898044 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 19 11:47:24 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jun 19 11:47:24 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jun 19 11:47:24 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jun 19 11:47:24 oak-gw06 kernel: Lustre: oak-OST0002-osc-ffff88041b99c000: Connection to oak-OST0002 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 19 11:47:24 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jun 19 11:47:24 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jun 19 11:47:30 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1497898044/real 0] req@ffff88039bba1b00 x1566264203573296/t0(0) o8->oak-OST000a-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497898050 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 19 11:47:30 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jun 19 11:48:14 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497898094/real 1497898094] req@ffff880226a10300 x1566264203576752/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497898105 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 19 11:48:14 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 19 11:49:04 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497898144/real 1497898144] req@ffff88030189db00 x1566264203578064/t0(0) o8->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497898160 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 19 11:49:04 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 19 11:49:54 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497898194/real 1497898194] req@ffff88030189e400 x1566264203579216/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497898215 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 19 11:49:54 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 19 11:50:44 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497898244/real 1497898244] req@ffff8804132c4f00 x1566264203580464/t0(0) o8->oak-OST0002-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497898270 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 19 11:50:44 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 19 11:51:34 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497898294/real 1497898294] req@ffff880292283000 x1566264203583824/t0(0) o8->oak-OST0006-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497898325 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 19 11:51:34 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 19 11:52:24 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497898344/real 1497898344] req@ffff8801fce12d00 x1566264203587568/t0(0) o8->oak-OST0006-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497898380 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 19 11:52:24 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jun 19 11:54:04 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497898444/real 1497898444] req@ffff88030189f600 x1566264203594976/t0(0) o8->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497898490 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 19 11:54:04 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 35 previous similar messages Jun 19 11:56:34 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497898594/real 1497898594] req@ffff880095733000 x1566264203606032/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1497898649 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 19 11:56:34 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 53 previous similar messages Jun 19 12:07:35 oak-gw06 kernel: Lustre: oak-OST000e-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jun 19 12:07:37 oak-gw06 kernel: Lustre: oak-OST0022-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jun 19 12:07:39 oak-gw06 kernel: Lustre: oak-OST0018-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jun 19 12:07:39 oak-gw06 kernel: Lustre: Skipped 4 previous similar messages Jun 19 12:07:41 oak-gw06 kernel: Lustre: oak-OST0004-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jun 19 12:07:41 oak-gw06 kernel: Lustre: Skipped 3 previous similar messages Jun 19 12:07:45 oak-gw06 kernel: Lustre: oak-OST001e-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jun 19 12:07:45 oak-gw06 kernel: Lustre: Skipped 6 previous similar messages Jun 19 14:02:56 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497906169/real 1497906169] req@ffff880310f59500 x1566264203848496/t0(0) o400->MGC10.0.2.51@o2ib5@10.0.2.51@o2ib5:26/25 lens 224/224 e 0 to 1 dl 1497906176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 19 14:02:56 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 71 previous similar messages Jun 19 14:02:56 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jun 19 14:03:57 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497906226/real 1497906226] req@ffff8801eab6a400 x1566264203849728/t0(0) o250->MGC10.0.2.51@o2ib5@10.0.2.51@o2ib5:26/25 lens 520/544 e 0 to 1 dl 1497906237 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 19 14:03:57 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jun 19 14:05:02 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497906276/real 1497906302] req@ffff88015a12f900 x1566264203850960/t0(0) o400->oak-MDT0000-mdc-ffff88041b99c000@10.0.2.51@o2ib5:12/10 lens 224/224 e 0 to 1 dl 1497906944 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 19 14:05:02 oak-gw06 kernel: Lustre: oak-MDT0000-mdc-ffff88041b99c000: Connection to oak-MDT0000 (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 19 14:05:02 oak-gw06 kernel: Lustre: Skipped 15 previous similar messages Jun 19 14:05:02 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jun 19 14:07:23 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497906427/real 1497906427] req@ffff8803e57e6400 x1566264203854608/t0(0) o38->oak-MDT0000-mdc-ffff88041b99c000@10.0.2.52@o2ib5:12/10 lens 520/544 e 0 to 1 dl 1497906443 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 19 14:07:23 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jun 19 14:08:47 oak-gw06 kernel: Lustre: Evicted from MGS (at 10.0.2.51@o2ib5) after server handle changed from 0xf0d4e97d03bcc4dc to 0xbfb40de81fe96ca0 Jun 19 14:08:47 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jun 19 14:13:57 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497906169/real 1497906169] req@ffff880310f5a100 x1566264203848512/t0(0) o400->oak-MDT0000-mdc-ffff88041b99c000@10.0.2.52@o2ib5:12/10 lens 224/224 e 0 to 1 dl 1497906837 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 19 14:13:57 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jun 19 14:22:34 oak-gw06 kernel: Lustre: oak-MDT0000-mdc-ffff88041b99c000: Connection restored to 10.0.2.52@o2ib5 (at 10.0.2.52@o2ib5) Jun 23 15:38:22 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1498257502/real 1498257502] req@ffff8804147df600 x1566264213138000/t0(0) o400->MGC10.0.2.51@o2ib5@10.0.2.51@o2ib5:26/25 lens 224/224 e 0 to 1 dl 1498257509 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 23 15:38:22 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jun 23 15:38:22 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jun 23 15:38:47 oak-gw06 kernel: Lustre: Evicted from MGS (at MGC10.0.2.51@o2ib5_1) after server handle changed from 0xbfb40de81fe96ca0 to 0x67c165d345e1b053 Jun 23 15:38:47 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to MGC10.0.2.51@o2ib5_1 (at 10.0.2.52@o2ib5) Jun 23 15:58:04 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1498258677/real 1498258677] req@ffff88013f5e7c00 x1566264213171248/t0(0) o400->MGC10.0.2.51@o2ib5@10.0.2.52@o2ib5:26/25 lens 224/224 e 0 to 1 dl 1498258684 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 23 15:58:04 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 23 15:58:04 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.52@o2ib5) was lost; in progress operations using this service will fail Jun 23 15:58:29 oak-gw06 kernel: Lustre: Evicted from MGS (at 10.0.2.51@o2ib5) after server handle changed from 0x67c165d345e1b053 to 0x5c44f40f6a1b0dbe Jun 23 15:58:29 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jun 23 17:38:54 oak-gw06 kernel: LustreError: 11-0: oak-OST0008-osc-ffff88041b99c000: operation obd_ping to node 10.0.2.101@o2ib5 failed: rc = -107 Jun 23 17:38:54 oak-gw06 kernel: Lustre: oak-OST0002-osc-ffff88041b99c000: Connection to oak-OST0002 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 23 17:38:54 oak-gw06 kernel: Lustre: Skipped 2 previous similar messages Jun 23 17:38:54 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Jun 23 17:39:51 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1498264784/real 1498264784] req@ffff8802f2737300 x1566264213344016/t0(0) o400->oak-OST0024-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1498264791 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 23 17:39:51 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 23 17:39:51 oak-gw06 kernel: Lustre: oak-OST0024-osc-ffff88041b99c000: Connection to oak-OST0024 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 23 17:39:51 oak-gw06 kernel: Lustre: Skipped 17 previous similar messages Jun 23 17:39:55 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1498264784/real 1498264784] req@ffff8802f2737900 x1566264213344048/t0(0) o8->oak-OST0026-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1498264795 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 23 17:39:57 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1498264791/real 1498264791] req@ffff8801871f4600 x1566264213344112/t0(0) o8->oak-OST0024-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1498264797 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 23 17:39:57 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jun 23 17:40:27 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1498264816/real 1498264816] req@ffff8801871f4600 x1566264213344416/t0(0) o8->oak-OST0010-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1498264827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 23 17:40:52 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1498264841/real 1498264841] req@ffff88036af57300 x1566264213345440/t0(0) o8->oak-OST0024-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1498264852 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 23 17:40:52 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jun 23 17:41:21 oak-gw06 kernel: Lustre: oak-OST0028-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 23 17:41:40 oak-gw06 kernel: Lustre: oak-OST0014-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 23 17:41:40 oak-gw06 kernel: Lustre: Skipped 15 previous similar messages Jun 23 17:42:05 oak-gw06 kernel: Lustre: oak-OST0020-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 23 17:43:41 oak-gw06 kernel: Lustre: oak-OST0024-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 23 17:43:41 oak-gw06 kernel: Lustre: Skipped 2 previous similar messages Jun 23 18:19:17 oak-gw06 kernel: LustreError: 11-0: oak-OST0006-osc-ffff88041b99c000: operation obd_ping to node 10.0.2.102@o2ib5 failed: rc = -107 Jun 23 18:19:17 oak-gw06 kernel: Lustre: oak-OST0000-osc-ffff88041b99c000: Connection to oak-OST0000 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 23 18:19:17 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jun 23 18:19:17 oak-gw06 kernel: LustreError: Skipped 21 previous similar messages Jun 23 18:19:42 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1498267182/real 1498267182] req@ffff880075b2cc00 x1566264213411120/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1498267188 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 23 18:19:42 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jun 23 18:20:21 oak-gw06 kernel: Lustre: oak-OST001c-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jun 23 18:20:24 oak-gw06 kernel: Lustre: oak-OST0024-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jun 23 18:20:29 oak-gw06 kernel: Lustre: oak-OST0020-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jun 23 18:20:29 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jun 23 18:20:46 oak-gw06 kernel: Lustre: oak-OST000e-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jun 23 18:20:59 oak-gw06 kernel: Lustre: oak-OST000c-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jun 23 18:20:59 oak-gw06 kernel: Lustre: Skipped 8 previous similar messages Jun 23 18:21:11 oak-gw06 kernel: Lustre: oak-OST0018-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jun 23 18:21:11 oak-gw06 kernel: Lustre: Skipped 2 previous similar messages Jun 23 18:23:59 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1498267432/real 1498267432] req@ffff8802b770c900 x1566264213418528/t0(0) o400->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1498267439 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 23 18:23:59 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1498267432/real 1498267432] req@ffff8802b770ea00 x1566264213418464/t0(0) o400->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1498267439 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 23 18:23:59 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1498267432/real 1498267432] req@ffff8802b770f900 x1566264213418496/t0(0) o400->oak-OST0003-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1498267439 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 23 18:23:59 oak-gw06 kernel: Lustre: oak-OST0015-osc-ffff88041b99c000: Connection to oak-OST0015 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 23 18:23:59 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jun 23 18:23:59 oak-gw06 kernel: Lustre: Skipped 18 previous similar messages Jun 23 18:23:59 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jun 23 18:23:59 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jun 23 18:24:49 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1498267489/real 1498267489] req@ffff8803cc761e00 x1566264213420208/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1498267500 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 23 18:24:49 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jun 23 18:29:06 oak-gw06 kernel: Lustre: oak-OST0025-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jun 23 18:29:06 oak-gw06 kernel: Lustre: Skipped 4 previous similar messages Jun 23 18:29:38 oak-gw06 kernel: Lustre: oak-OST0009-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Jun 23 18:29:38 oak-gw06 kernel: Lustre: Skipped 11 previous similar messages Jun 23 19:15:39 oak-gw06 kernel: LustreError: 11-0: oak-OST0007-osc-ffff88041b99c000: operation obd_ping to node 10.0.2.101@o2ib5 failed: rc = -107 Jun 23 19:15:39 oak-gw06 kernel: Lustre: oak-OST000d-osc-ffff88041b99c000: Connection to oak-OST000d (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 23 19:15:39 oak-gw06 kernel: Lustre: Skipped 20 previous similar messages Jun 23 19:15:39 oak-gw06 kernel: LustreError: Skipped 18 previous similar messages Jun 23 19:16:04 oak-gw06 kernel: LustreError: 11-0: oak-OST0015-osc-ffff88041b99c000: operation obd_ping to node 10.0.2.101@o2ib5 failed: rc = -107 Jun 23 19:16:04 oak-gw06 kernel: Lustre: oak-OST001d-osc-ffff88041b99c000: Connection to oak-OST001d (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 23 19:16:04 oak-gw06 kernel: Lustre: Skipped 17 previous similar messages Jun 23 19:16:04 oak-gw06 kernel: LustreError: Skipped 4 previous similar messages Jun 23 19:16:04 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1498270564/real 1498270564] req@ffff88040c9d8f00 x1566264213504096/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1498270570 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 23 19:16:04 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Jun 23 19:16:35 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1498270589/real 1498270589] req@ffff880321758900 x1566264213505120/t0(0) o8->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1498270595 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 23 19:16:35 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jun 23 19:19:16 oak-gw06 kernel: Lustre: oak-OST000b-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 23 19:19:16 oak-gw06 kernel: Lustre: Skipped 8 previous similar messages Jun 23 19:19:24 oak-gw06 kernel: Lustre: oak-OST000f-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 23 19:19:24 oak-gw06 kernel: Lustre: Skipped 4 previous similar messages Jun 23 19:19:42 oak-gw06 kernel: Lustre: oak-OST001b-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jun 23 19:19:42 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Jul 7 12:40:51 oak-gw06 kernel: Lustre: DEBUG MARKER: Fri Jul 7 12:40:51 2017 Jul 11 05:02:29 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3126:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 9 seconds Jul 11 05:02:29 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3189:kiblnd_check_conns()) Timed out RDMA with 10.0.2.102@o2ib5 (59): c: 0, oc: 0, rc: 8 Jul 11 05:02:29 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1499774489/real 1499774549] req@ffff880399575500 x1566264255853568/t0(0) o400->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1499775245 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jul 11 05:02:29 oak-gw06 kernel: Lustre: oak-OST0019-osc-ffff88041b99c000: Connection to oak-OST0019 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jul 11 05:02:29 oak-gw06 kernel: Lustre: Skipped 2 previous similar messages Jul 11 05:02:29 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 11 05:02:35 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499774549/real 0] req@ffff880096dd1500 x1566264255855264/t0(0) o8->oak-OST000d-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499774555 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 05:02:35 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 11 05:04:08 oak-gw06 kernel: Lustre: 1769:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1499774489/real 1499774648] req@ffff880399574000 x1566264255853632/t0(0) o400->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1499775245 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jul 11 05:04:08 oak-gw06 kernel: Lustre: oak-OST0025-osc-ffff88041b99c000: Connection to oak-OST0025 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jul 11 05:04:08 oak-gw06 kernel: Lustre: Skipped 6 previous similar messages Jul 11 05:04:08 oak-gw06 kernel: Lustre: 1769:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 11 05:04:14 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499774648/real 0] req@ffff8802ee698c00 x1566264255857136/t0(0) o8->oak-OST0029-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499774654 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 05:04:14 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 11 05:05:46 oak-gw06 kernel: Lustre: 1767:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1499774514/real 1499774746] req@ffff8800afd25500 x1566264255853984/t0(0) o400->oak-OST000b-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1499775270 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jul 11 05:05:46 oak-gw06 kernel: Lustre: oak-OST0007-osc-ffff88041b99c000: Connection to oak-OST0007 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jul 11 05:05:46 oak-gw06 kernel: Lustre: Skipped 6 previous similar messages Jul 11 05:05:46 oak-gw06 kernel: Lustre: 1767:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 11 05:07:25 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1499774514/real 1499774845] req@ffff8802e7eb4300 x1566264255854240/t0(0) o400->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1499775270 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jul 11 05:07:25 oak-gw06 kernel: Lustre: oak-OST0001-osc-ffff88041b99c000: Connection to oak-OST0001 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jul 11 05:07:25 oak-gw06 kernel: Lustre: Skipped 2 previous similar messages Jul 11 05:07:25 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 11 05:09:03 oak-gw06 kernel: Lustre: 1769:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1499774514/real 1499774943] req@ffff8802e7eb4900 x1566264255854272/t0(0) o400->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1499775270 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jul 11 05:09:03 oak-gw06 kernel: Lustre: 1769:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 11 05:10:42 oak-gw06 kernel: Lustre: 1767:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1499774539/real 1499775042] req@ffff8801beed2400 x1566264255854880/t0(0) o400->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1499775295 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jul 11 05:10:42 oak-gw06 kernel: Lustre: 1767:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 11 05:13:01 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499775170/real 0] req@ffff88023cb0e700 x1566264255864480/t0(0) o8->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499775181 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 05:13:01 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 50 previous similar messages Jul 11 05:17:34 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499774698/real 0] req@ffff8802b770de00 x1566264255857760/t0(0) o400->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1499775454 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 05:17:34 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 101 previous similar messages Jul 11 05:28:01 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499776070/real 0] req@ffff88023cb0cc00 x1566264255878144/t0(0) o8->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499776081 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 05:28:01 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Jul 11 05:39:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499776745/real 0] req@ffff88015a718300 x1566264255889744/t0(0) o8->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499776761 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 05:39:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 39 previous similar messages Jul 11 05:49:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499777345/real 0] req@ffff88027e930000 x1566264255900112/t0(0) o8->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499777371 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 05:49:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 47 previous similar messages Jul 11 05:59:41 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499777945/real 0] req@ffff88036bed8600 x1566264255910480/t0(0) o8->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499777981 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 05:59:41 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 47 previous similar messages Jul 11 06:10:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499778620/real 0] req@ffff880325e17300 x1566264255921952/t0(0) o8->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499778631 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 06:10:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 55 previous similar messages Jul 11 06:20:41 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499779220/real 0] req@ffff8800a96b9800 x1566264255932288/t0(0) o8->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499779241 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 06:20:41 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 47 previous similar messages Jul 11 06:30:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499779820/real 0] req@ffff8803307ce700 x1566264255942656/t0(0) o8->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499779856 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 06:30:56 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 52 previous similar messages Jul 11 06:42:01 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499780495/real 0] req@ffff88000654c000 x1566264255954592/t0(0) o8->oak-OST0019-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499780521 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 06:42:01 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 50 previous similar messages Jul 11 06:52:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499781095/real 0] req@ffff88025e0e2100 x1566264255965136/t0(0) o8->oak-OST0027-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499781131 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 06:52:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 50 previous similar messages Jul 11 07:02:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499781695/real 0] req@ffff8801ebb00c00 x1566264255975328/t0(0) o8->oak-OST0019-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499781741 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 07:02:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 47 previous similar messages Jul 11 07:13:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499782370/real 0] req@ffff8803f62ccf00 x1566264255986848/t0(0) o8->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499782411 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 07:13:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 52 previous similar messages Jul 11 07:23:45 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499782970/real 0] req@ffff8803f62cf900 x1566264255997216/t0(0) o8->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499783025 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 07:23:45 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 49 previous similar messages Jul 11 07:34:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499783645/real 0] req@ffff880009b26700 x1566264256008912/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499783661 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 07:34:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 53 previous similar messages Jul 11 07:44:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499784245/real 0] req@ffff88042eeae700 x1566264256019280/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499784276 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 07:44:36 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 47 previous similar messages Jul 11 07:54:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499784845/real 0] req@ffff8802bb2f8c00 x1566264256029792/t0(0) o8->oak-OST000d-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499784886 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 07:54:46 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 47 previous similar messages Jul 11 08:06:15 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499785520/real 0] req@ffff88003ab99200 x1566264256041408/t0(0) o8->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499785575 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 08:06:15 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 55 previous similar messages Jul 11 08:16:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499786120/real 0] req@ffff880204b82d00 x1566264256051776/t0(0) o8->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499786176 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 08:16:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 52 previous similar messages Jul 11 08:27:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499786820/real 0] req@ffff8803bfc04c00 x1566264256064160/t0(0) o8->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499786836 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 08:27:16 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 50 previous similar messages Jul 11 08:37:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499787395/real 0] req@ffff8803473ad200 x1566264256073936/t0(0) o8->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499787441 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 08:37:21 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 52 previous similar messages Jul 11 08:47:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499787995/real 0] req@ffff8802b770c900 x1566264256084208/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499788051 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 08:47:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 47 previous similar messages Jul 11 08:58:01 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499788670/real 0] req@ffff88005e200f00 x1566264256096000/t0(0) o8->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499788681 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 08:58:01 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 50 previous similar messages Jul 11 09:08:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499789270/real 0] req@ffff8801b5f60600 x1566264256106336/t0(0) o8->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499789291 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 09:08:11 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 49 previous similar messages Jul 11 09:18:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499789870/real 0] req@ffff880306f3f000 x1566264256116704/t0(0) o8->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499789906 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 09:18:26 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 50 previous similar messages Jul 11 09:29:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499790545/real 0] req@ffff88000c0bcc00 x1566264256128640/t0(0) o8->oak-OST0019-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499790571 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 09:29:31 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 50 previous similar messages Jul 11 09:39:41 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499791145/real 0] req@ffff88003f865500 x1566264256139104/t0(0) o8->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499791181 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 09:39:41 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 50 previous similar messages Jul 11 09:49:51 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499791745/real 0] req@ffff880399574300 x1566264256149472/t0(0) o8->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499791791 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 09:49:51 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 47 previous similar messages Jul 11 10:01:01 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1499792420/real 0] req@ffff8803737bad00 x1566264256160896/t0(0) o8->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499792461 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 11 10:01:01 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 52 previous similar messages Jul 11 10:11:10 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1499793070/real 1499793070] req@ffff880062d9c000 x1566264256172848/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499793126 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jul 11 10:11:10 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 68 previous similar messages Jul 11 10:21:10 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1499793670/real 1499793670] req@ffff880207b29e00 x1566264256189744/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1499793711 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jul 11 10:21:10 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 251 previous similar messages Jul 11 10:32:08 oak-gw06 kernel: Lustre: oak-OST0007-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jul 11 10:32:08 oak-gw06 kernel: Lustre: Skipped 4 previous similar messages Jul 11 10:32:12 oak-gw06 kernel: Lustre: oak-OST001f-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jul 11 10:32:12 oak-gw06 kernel: Lustre: Skipped 6 previous similar messages Jul 18 02:46:01 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Jul 18 02:46:01 oak-gw06 kernel: CPU: 0 PID: 14579 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 02:46:01 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 02:46:01 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 02:46:01 oak-gw06 kernel: 00000000000080d0 00000000d4efb18b ffff880190a7f858 ffffffff8168662f Jul 18 02:46:01 oak-gw06 kernel: ffff880190a7f8e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Jul 18 02:46:01 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff880190a7f8e8 00000000d4efb18b Jul 18 02:46:01 oak-gw06 kernel: Call Trace: Jul 18 02:46:01 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 02:46:01 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 02:46:01 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 02:46:01 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 02:46:01 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Jul 18 02:46:01 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Jul 18 02:46:01 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 02:46:01 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 02:46:01 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 02:46:01 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 02:46:01 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 02:46:01 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 02:46:01 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 02:46:01 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 02:46:01 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 02:46:01 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 02:46:01 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 02:46:01 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 02:46:01 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 02:46:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 02:46:01 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 02:46:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 02:46:01 oak-gw06 kernel: Mem-Info: Jul 18 02:46:01 oak-gw06 kernel: active_anon:40749 inactive_anon:35867 isolated_anon:0#012 active_file:1749461 inactive_file:387450 isolated_file:10#012 unevictable:0 dirty:13751 writeback:5963 unstable:0#012 slab_reclaimable:52667 slab_unreclaimable:897508#012 mapped:9949 shmem:30808 pagetables:1647 bounce:0#012 free:729629 free_pcp:2073 free_cma:0 Jul 18 02:46:01 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 02:46:01 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 02:46:01 oak-gw06 kernel: Node 0 DMA32 free:670248kB min:11976kB low:14968kB high:17964kB active_anon:23320kB inactive_anon:27040kB active_file:1103312kB inactive_file:271364kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:10812kB writeback:4844kB mapped:4644kB shmem:24540kB slab_reclaimable:38648kB slab_unreclaimable:636496kB kernel_stack:1008kB pagetables:1084kB unstable:0kB bounce:0kB free_pcp:4312kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 02:46:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 02:46:01 oak-gw06 kernel: Node 0 Normal free:2263788kB min:55536kB low:69420kB high:83304kB active_anon:139676kB inactive_anon:116428kB active_file:5894532kB inactive_file:1249300kB unevictable:0kB isolated(anon):0kB isolated(file):40kB present:13631488kB managed:13367060kB mlocked:0kB dirty:35364kB writeback:16200kB mapped:35152kB shmem:98692kB slab_reclaimable:172020kB slab_unreclaimable:2949568kB kernel_stack:4672kB pagetables:5504kB unstable:0kB bounce:0kB free_pcp:4252kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 02:46:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 02:46:01 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 02:46:01 oak-gw06 kernel: Node 0 DMA32: 2932*4kB (UEM) 6254*8kB (UEM) 6579*16kB (UEM) 9168*32kB (UEM) 2623*64kB (UEM) 353*128kB (UEM) 9*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 675760kB Jul 18 02:46:01 oak-gw06 kernel: Node 0 Normal: 5863*4kB (UEM) 29001*8kB (UEM) 37317*16kB (UEM) 31450*32kB (UEM) 5617*64kB (UEM) 461*128kB (UEM) 4*256kB (EM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2278452kB Jul 18 02:46:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 02:46:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 02:46:01 oak-gw06 kernel: 2043405 total pagecache pages Jul 18 02:46:01 oak-gw06 kernel: 0 pages in swap cache Jul 18 02:46:01 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 02:46:01 oak-gw06 kernel: Free swap = 4194300kB Jul 18 02:46:01 oak-gw06 kernel: Total swap = 4194300kB Jul 18 02:46:01 oak-gw06 kernel: 4194203 pages RAM Jul 18 02:46:01 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 02:46:01 oak-gw06 kernel: 127313 pages reserved Jul 18 02:46:01 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Jul 18 02:46:01 oak-gw06 kernel: CPU: 0 PID: 14579 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 02:46:01 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 02:46:01 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 02:46:01 oak-gw06 kernel: 00000000000080d0 00000000d4efb18b ffff880190a7f808 ffffffff8168662f Jul 18 02:46:01 oak-gw06 kernel: ffff880190a7f898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Jul 18 02:46:01 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880190a7f868 00000000d4efb18b Jul 18 02:46:01 oak-gw06 kernel: Call Trace: Jul 18 02:46:01 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 02:46:01 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 02:46:01 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Jul 18 02:46:01 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 02:46:01 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 02:46:01 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Jul 18 02:46:01 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Jul 18 02:46:01 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Jul 18 02:46:01 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Jul 18 02:46:01 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 02:46:01 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 02:46:01 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 02:46:01 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 02:46:01 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 02:46:01 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 02:46:01 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 02:46:01 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 02:46:01 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 02:46:01 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 02:46:01 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 02:46:01 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 02:46:01 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 02:46:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 02:46:01 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 02:46:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 02:46:01 oak-gw06 kernel: Mem-Info: Jul 18 02:46:01 oak-gw06 kernel: active_anon:39855 inactive_anon:36826 isolated_anon:0#012 active_file:1749396 inactive_file:369016 isolated_file:10#012 unevictable:0 dirty:14497 writeback:5601 unstable:0#012 slab_reclaimable:52667 slab_unreclaimable:895956#012 mapped:9949 shmem:30808 pagetables:1647 bounce:0#012 free:752190 free_pcp:2180 free_cma:0 Jul 18 02:46:01 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 02:46:01 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 02:46:01 oak-gw06 kernel: Node 0 DMA32 free:682364kB min:11976kB low:14968kB high:17964kB active_anon:23320kB inactive_anon:27040kB active_file:1103312kB inactive_file:261200kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:10812kB writeback:4868kB mapped:4644kB shmem:24540kB slab_reclaimable:38648kB slab_unreclaimable:635872kB kernel_stack:1008kB pagetables:1084kB unstable:0kB bounce:0kB free_pcp:4256kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 02:46:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 02:46:01 oak-gw06 kernel: Node 0 Normal free:2301492kB min:55536kB low:69420kB high:83304kB active_anon:136100kB inactive_anon:120264kB active_file:5894272kB inactive_file:1228240kB unevictable:0kB isolated(anon):0kB isolated(file):40kB present:13631488kB managed:13367060kB mlocked:0kB dirty:50108kB writeback:16588kB mapped:35152kB shmem:98692kB slab_reclaimable:172020kB slab_unreclaimable:2947936kB kernel_stack:4672kB pagetables:5504kB unstable:0kB bounce:0kB free_pcp:4832kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 02:46:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 02:46:01 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 02:46:01 oak-gw06 kernel: Node 0 DMA32: 1076*4kB (UEM) 6577*8kB (UEM) 7279*16kB (UEM) 9186*32kB (UEM) 2625*64kB (UEM) 353*128kB (UEM) 9*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 682824kB Jul 18 02:46:01 oak-gw06 kernel: Node 0 Normal: 2800*4kB (UE) 27724*8kB (UEM) 39648*16kB (UEM) 31544*32kB (UEM) 5620*64kB (UEM) 461*128kB (UEM) 4*256kB (EM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2296480kB Jul 18 02:46:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 02:46:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 02:46:01 oak-gw06 kernel: 2006777 total pagecache pages Jul 18 02:46:01 oak-gw06 kernel: 0 pages in swap cache Jul 18 02:46:01 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 02:46:01 oak-gw06 kernel: Free swap = 4194300kB Jul 18 02:46:01 oak-gw06 kernel: Total swap = 4194300kB Jul 18 02:46:01 oak-gw06 kernel: 4194203 pages RAM Jul 18 02:46:01 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 02:46:01 oak-gw06 kernel: 127313 pages reserved Jul 18 03:26:03 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Jul 18 03:26:03 oak-gw06 kernel: CPU: 5 PID: 14608 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 03:26:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 03:26:03 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 03:26:03 oak-gw06 kernel: 00000000000080d0 00000000427bdb5e ffff8803d9cb7858 ffffffff8168662f Jul 18 03:26:03 oak-gw06 kernel: ffff8803d9cb78e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Jul 18 03:26:03 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff8803d9cb78e8 00000000427bdb5e Jul 18 03:26:03 oak-gw06 kernel: Call Trace: Jul 18 03:26:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 03:26:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 03:26:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 03:26:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 03:26:03 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Jul 18 03:26:03 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Jul 18 03:26:03 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 03:26:03 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 03:26:03 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 03:26:03 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 03:26:03 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 03:26:03 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 03:26:03 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 03:26:03 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 03:26:03 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 03:26:03 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 03:26:03 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 03:26:03 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 03:26:03 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 03:26:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 03:26:03 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 03:26:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 03:26:03 oak-gw06 kernel: Mem-Info: Jul 18 03:26:03 oak-gw06 kernel: active_anon:34883 inactive_anon:36859 isolated_anon:0#012 active_file:1021450 inactive_file:1167710 isolated_file:0#012 unevictable:0 dirty:20559 writeback:7806 unstable:0#012 slab_reclaimable:52426 slab_unreclaimable:897597#012 mapped:10055 shmem:30816 pagetables:1660 bounce:0#012 free:639341 free_pcp:678 free_cma:0 Jul 18 03:26:03 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 03:26:03 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 03:26:03 oak-gw06 kernel: Node 0 DMA32 free:469372kB min:11976kB low:14968kB high:17964kB active_anon:19228kB inactive_anon:27040kB active_file:705872kB inactive_file:846516kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:15832kB writeback:7052kB mapped:4664kB shmem:24540kB slab_reclaimable:38520kB slab_unreclaimable:640624kB kernel_stack:1008kB pagetables:1144kB unstable:0kB bounce:0kB free_pcp:1180kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 03:26:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 03:26:03 oak-gw06 kernel: Node 0 Normal free:2060028kB min:55536kB low:69420kB high:83304kB active_anon:120304kB inactive_anon:120396kB active_file:3379928kB inactive_file:3835056kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:63492kB writeback:25340kB mapped:35556kB shmem:98724kB slab_reclaimable:171184kB slab_unreclaimable:2950292kB kernel_stack:4688kB pagetables:5496kB unstable:0kB bounce:0kB free_pcp:1744kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 03:26:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 03:26:03 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 03:26:03 oak-gw06 kernel: Node 0 DMA32: 395*4kB (UEM) 2032*8kB (UEM) 7470*16kB (UEM) 4773*32kB (UEM) 2240*64kB (UEM) 266*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 467500kB Jul 18 03:26:03 oak-gw06 kernel: Node 0 Normal: 1277*4kB (UEM) 15569*8kB (UEM) 36201*16kB (UEM) 28989*32kB (UEM) 5917*64kB (UEM) 303*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2053996kB Jul 18 03:26:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 03:26:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 03:26:03 oak-gw06 kernel: 2099559 total pagecache pages Jul 18 03:26:03 oak-gw06 kernel: 0 pages in swap cache Jul 18 03:26:03 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 03:26:03 oak-gw06 kernel: Free swap = 4194300kB Jul 18 03:26:03 oak-gw06 kernel: Total swap = 4194300kB Jul 18 03:26:03 oak-gw06 kernel: 4194203 pages RAM Jul 18 03:26:03 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 03:26:03 oak-gw06 kernel: 127313 pages reserved Jul 18 03:26:03 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Jul 18 03:26:03 oak-gw06 kernel: CPU: 5 PID: 14608 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 03:26:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 03:26:03 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 03:26:03 oak-gw06 kernel: 00000000000080d0 00000000427bdb5e ffff8803d9cb7808 ffffffff8168662f Jul 18 03:26:03 oak-gw06 kernel: ffff8803d9cb7898 ffffffff81186ba0 0000000000000000 00000000ffffffff Jul 18 03:26:03 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803d9cb7868 00000000427bdb5e Jul 18 03:26:03 oak-gw06 kernel: Call Trace: Jul 18 03:26:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 03:26:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 03:26:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 03:26:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 03:26:03 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Jul 18 03:26:03 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Jul 18 03:26:03 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Jul 18 03:26:03 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Jul 18 03:26:03 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 03:26:03 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 03:26:03 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 03:26:03 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 03:26:03 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 03:26:03 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 03:26:03 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 03:26:03 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 03:26:03 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 03:26:03 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 03:26:03 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 03:26:03 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 03:26:03 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 03:26:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 03:26:03 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 03:26:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 03:26:03 oak-gw06 kernel: Mem-Info: Jul 18 03:26:03 oak-gw06 kernel: active_anon:34883 inactive_anon:36859 isolated_anon:0#012 active_file:1021450 inactive_file:1178745 isolated_file:0#012 unevictable:0 dirty:23627 writeback:9335 unstable:0#012 slab_reclaimable:52426 slab_unreclaimable:897701#012 mapped:10055 shmem:30816 pagetables:1660 bounce:0#012 free:627947 free_pcp:653 free_cma:0 Jul 18 03:26:03 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 03:26:03 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 03:26:03 oak-gw06 kernel: Node 0 DMA32 free:461876kB min:11976kB low:14968kB high:17964kB active_anon:19228kB inactive_anon:27040kB active_file:705872kB inactive_file:853232kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:16488kB writeback:6224kB mapped:4664kB shmem:24540kB slab_reclaimable:38520kB slab_unreclaimable:641040kB kernel_stack:1008kB pagetables:1144kB unstable:0kB bounce:0kB free_pcp:964kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 03:26:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 03:26:03 oak-gw06 kernel: Node 0 Normal free:2027524kB min:55536kB low:69420kB high:83304kB active_anon:120304kB inactive_anon:120396kB active_file:3379928kB inactive_file:3870416kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:66208kB writeback:26116kB mapped:35556kB shmem:98724kB slab_reclaimable:171184kB slab_unreclaimable:2949476kB kernel_stack:4688kB pagetables:5496kB unstable:0kB bounce:0kB free_pcp:2164kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 03:26:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 03:26:03 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 03:26:03 oak-gw06 kernel: Node 0 DMA32: 295*4kB (UEM) 999*8kB (UEM) 7486*16kB (UEM) 4785*32kB (UEM) 2240*64kB (UEM) 266*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 459476kB Jul 18 03:26:03 oak-gw06 kernel: Node 0 Normal: 1339*4kB (UEM) 11216*8kB (UEM) 36210*16kB (UEM) 28984*32kB (UEM) 5916*64kB (EM) 305*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2019596kB Jul 18 03:26:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 03:26:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 03:26:03 oak-gw06 kernel: 2063557 total pagecache pages Jul 18 03:26:03 oak-gw06 kernel: 0 pages in swap cache Jul 18 03:26:03 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 03:26:03 oak-gw06 kernel: Free swap = 4194300kB Jul 18 03:26:03 oak-gw06 kernel: Total swap = 4194300kB Jul 18 03:26:03 oak-gw06 kernel: 4194203 pages RAM Jul 18 03:26:03 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 03:26:03 oak-gw06 kernel: 127313 pages reserved Jul 18 06:56:16 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Jul 18 06:56:16 oak-gw06 kernel: CPU: 7 PID: 15019 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 06:56:16 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 06:56:16 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 06:56:16 oak-gw06 kernel: 00000000000080d0 000000004856e829 ffff88020d66b858 ffffffff8168662f Jul 18 06:56:16 oak-gw06 kernel: ffff88020d66b8e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Jul 18 06:56:16 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff88020d66b8e8 000000004856e829 Jul 18 06:56:16 oak-gw06 kernel: Call Trace: Jul 18 06:56:16 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 06:56:16 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 06:56:16 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 06:56:16 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 06:56:16 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Jul 18 06:56:16 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Jul 18 06:56:16 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 06:56:16 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 06:56:16 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 06:56:16 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 06:56:16 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 06:56:16 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 06:56:16 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 06:56:16 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 06:56:16 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 06:56:16 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 06:56:16 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 06:56:16 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 06:56:16 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 06:56:16 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 06:56:16 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 06:56:16 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 06:56:16 oak-gw06 kernel: Mem-Info: Jul 18 06:56:16 oak-gw06 kernel: active_anon:30936 inactive_anon:36856 isolated_anon:0#012 active_file:36391 inactive_file:2017697 isolated_file:0#012 unevictable:0 dirty:23452 writeback:7490 unstable:0#012 slab_reclaimable:51926 slab_unreclaimable:887116#012 mapped:10076 shmem:30808 pagetables:1638 bounce:0#012 free:755697 free_pcp:1530 free_cma:0 Jul 18 06:56:16 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 06:56:16 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 06:56:16 oak-gw06 kernel: Node 0 DMA32 free:537948kB min:11976kB low:14968kB high:17964kB active_anon:18860kB inactive_anon:27040kB active_file:21876kB inactive_file:1446996kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:17704kB writeback:6904kB mapped:4668kB shmem:24540kB slab_reclaimable:38168kB slab_unreclaimable:623300kB kernel_stack:976kB pagetables:1312kB unstable:0kB bounce:0kB free_pcp:2708kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 06:56:16 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 06:56:16 oak-gw06 kernel: Node 0 Normal free:2499820kB min:55536kB low:69420kB high:83304kB active_anon:104884kB inactive_anon:120384kB active_file:123688kB inactive_file:6589848kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:81148kB writeback:23056kB mapped:35636kB shmem:98692kB slab_reclaimable:169536kB slab_unreclaimable:2925148kB kernel_stack:4704kB pagetables:5240kB unstable:0kB bounce:0kB free_pcp:3100kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 06:56:16 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 06:56:16 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 06:56:16 oak-gw06 kernel: Node 0 DMA32: 761*4kB (UEM) 5855*8kB (UEM) 4059*16kB (UEM) 1872*32kB (UEM) 1737*64kB (UEM) 1661*128kB (UM) 244*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 560972kB Jul 18 06:56:16 oak-gw06 kernel: Node 0 Normal: 3507*4kB (EM) 28375*8kB (UEM) 13793*16kB (UEM) 8661*32kB (UEM) 8894*64kB (UEM) 7888*128kB (UEM) 984*256kB (EM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2569652kB Jul 18 06:56:16 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 06:56:16 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 06:56:16 oak-gw06 kernel: 1994253 total pagecache pages Jul 18 06:56:16 oak-gw06 kernel: 0 pages in swap cache Jul 18 06:56:16 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 06:56:16 oak-gw06 kernel: Free swap = 4194300kB Jul 18 06:56:16 oak-gw06 kernel: Total swap = 4194300kB Jul 18 06:56:16 oak-gw06 kernel: 4194203 pages RAM Jul 18 06:56:16 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 06:56:16 oak-gw06 kernel: 127313 pages reserved Jul 18 06:56:16 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Jul 18 06:56:16 oak-gw06 kernel: CPU: 7 PID: 15019 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 06:56:16 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 06:56:16 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 06:56:16 oak-gw06 kernel: 00000000000080d0 000000004856e829 ffff88020d66b808 ffffffff8168662f Jul 18 06:56:16 oak-gw06 kernel: ffff88020d66b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Jul 18 06:56:16 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88020d66b868 000000004856e829 Jul 18 06:56:16 oak-gw06 kernel: Call Trace: Jul 18 06:56:16 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 06:56:16 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 06:56:16 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 06:56:16 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 06:56:16 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Jul 18 06:56:16 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Jul 18 06:56:16 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Jul 18 06:56:16 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Jul 18 06:56:16 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 06:56:16 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 06:56:16 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 06:56:16 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 06:56:16 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 06:56:16 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 06:56:16 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 06:56:16 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 06:56:16 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 06:56:16 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 06:56:16 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 06:56:16 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 06:56:16 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 06:56:16 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 06:56:16 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 06:56:16 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 06:56:16 oak-gw06 kernel: Mem-Info: Jul 18 06:56:16 oak-gw06 kernel: active_anon:30936 inactive_anon:36856 isolated_anon:0#012 active_file:36391 inactive_file:1933597 isolated_file:0#012 unevictable:0 dirty:22062 writeback:7059 unstable:0#012 slab_reclaimable:51926 slab_unreclaimable:886832#012 mapped:10076 shmem:30808 pagetables:1638 bounce:0#012 free:840014 free_pcp:898 free_cma:0 Jul 18 06:56:16 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 06:56:16 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 06:56:16 oak-gw06 kernel: Node 0 DMA32 free:605352kB min:11976kB low:14968kB high:17964kB active_anon:18860kB inactive_anon:27040kB active_file:21876kB inactive_file:1382700kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:15464kB writeback:6344kB mapped:4668kB shmem:24540kB slab_reclaimable:38168kB slab_unreclaimable:623300kB kernel_stack:976kB pagetables:1312kB unstable:0kB bounce:0kB free_pcp:1600kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 06:56:16 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 06:56:16 oak-gw06 kernel: Node 0 Normal free:2729312kB min:55536kB low:69420kB high:83304kB active_anon:105404kB inactive_anon:120384kB active_file:123688kB inactive_file:6360528kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:73388kB writeback:23056kB mapped:35636kB shmem:98692kB slab_reclaimable:169536kB slab_unreclaimable:2924012kB kernel_stack:4704kB pagetables:5240kB unstable:0kB bounce:0kB free_pcp:1784kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 06:56:16 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 06:56:16 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 06:56:16 oak-gw06 kernel: Node 0 DMA32: 2414*4kB (UEM) 8755*8kB (UEM) 4449*16kB (UEM) 1902*32kB (UEM) 1774*64kB (UEM) 1665*128kB (UM) 244*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 600864kB Jul 18 06:56:16 oak-gw06 kernel: Node 0 Normal: 9697*4kB (UEM) 39540*8kB (UEM) 15620*16kB (UEM) 8741*32kB (UEM) 9093*64kB (UEM) 7889*128kB (UEM) 984*256kB (EM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2728388kB Jul 18 06:56:16 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 06:56:16 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 06:56:16 oak-gw06 kernel: 2000762 total pagecache pages Jul 18 06:56:16 oak-gw06 kernel: 0 pages in swap cache Jul 18 06:56:16 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 06:56:16 oak-gw06 kernel: Free swap = 4194300kB Jul 18 06:56:16 oak-gw06 kernel: Total swap = 4194300kB Jul 18 06:56:16 oak-gw06 kernel: 4194203 pages RAM Jul 18 06:56:16 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 06:56:16 oak-gw06 kernel: 127313 pages reserved Jul 18 07:01:16 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Jul 18 07:01:16 oak-gw06 kernel: CPU: 7 PID: 15019 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 07:01:16 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 07:01:16 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 07:01:16 oak-gw06 kernel: 00000000000080d0 000000004856e829 ffff88020d66b858 ffffffff8168662f Jul 18 07:01:16 oak-gw06 kernel: ffff88020d66b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Jul 18 07:01:16 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88020d66b8b8 000000004856e829 Jul 18 07:01:16 oak-gw06 kernel: Call Trace: Jul 18 07:01:16 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 07:01:16 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 07:01:16 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 07:01:16 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 07:01:16 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Jul 18 07:01:16 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Jul 18 07:01:16 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 07:01:16 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 07:01:16 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 07:01:16 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 07:01:16 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 07:01:16 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 07:01:16 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 07:01:16 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 07:01:16 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 07:01:16 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 07:01:16 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 07:01:16 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 07:01:16 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 07:01:16 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 07:01:16 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 07:01:16 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 07:01:16 oak-gw06 kernel: Mem-Info: Jul 18 07:01:16 oak-gw06 kernel: active_anon:30937 inactive_anon:36856 isolated_anon:0#012 active_file:36393 inactive_file:2005435 isolated_file:0#012 unevictable:0 dirty:22817 writeback:8061 unstable:0#012 slab_reclaimable:51926 slab_unreclaimable:893852#012 mapped:10092 shmem:30808 pagetables:1639 bounce:0#012 free:755065 free_pcp:2136 free_cma:0 Jul 18 07:01:16 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 07:01:16 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 07:01:16 oak-gw06 kernel: Node 0 DMA32 free:539716kB min:11976kB low:14968kB high:17964kB active_anon:18848kB inactive_anon:27040kB active_file:21876kB inactive_file:1433216kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:15388kB writeback:6680kB mapped:4668kB shmem:24540kB slab_reclaimable:38168kB slab_unreclaimable:627572kB kernel_stack:976kB pagetables:1316kB unstable:0kB bounce:0kB free_pcp:4004kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 07:01:16 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 07:01:16 oak-gw06 kernel: Node 0 Normal free:2459744kB min:55536kB low:69420kB high:83304kB active_anon:104900kB inactive_anon:120384kB active_file:123696kB inactive_file:6598000kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:75536kB writeback:23624kB mapped:35700kB shmem:98692kB slab_reclaimable:169536kB slab_unreclaimable:2945372kB kernel_stack:4688kB pagetables:5240kB unstable:0kB bounce:0kB free_pcp:4004kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 07:01:16 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 07:01:16 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 07:01:16 oak-gw06 kernel: Node 0 DMA32: 5240*4kB (UEM) 6117*8kB (UEM) 3533*16kB (UEM) 1293*32kB (UEM) 1320*64kB (UEM) 1770*128kB (UM) 236*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 539256kB Jul 18 07:01:16 oak-gw06 kernel: Node 0 Normal: 22675*4kB (UEM) 26932*8kB (UEM) 12163*16kB (UEM) 5921*32kB (UEM) 8908*64kB (EM) 7835*128kB (EM) 762*256kB (EM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2458300kB Jul 18 07:01:16 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 07:01:16 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 07:01:16 oak-gw06 kernel: 2059167 total pagecache pages Jul 18 07:01:16 oak-gw06 kernel: 0 pages in swap cache Jul 18 07:01:16 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 07:01:16 oak-gw06 kernel: Free swap = 4194300kB Jul 18 07:01:16 oak-gw06 kernel: Total swap = 4194300kB Jul 18 07:01:16 oak-gw06 kernel: 4194203 pages RAM Jul 18 07:01:16 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 07:01:16 oak-gw06 kernel: 127313 pages reserved Jul 18 07:01:17 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Jul 18 07:01:17 oak-gw06 kernel: CPU: 7 PID: 15019 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 07:01:17 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 07:01:17 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 07:01:17 oak-gw06 kernel: 00000000000080d0 000000004856e829 ffff88020d66b808 ffffffff8168662f Jul 18 07:01:17 oak-gw06 kernel: ffff88020d66b898 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Jul 18 07:01:17 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff88020d66b898 000000004856e829 Jul 18 07:01:17 oak-gw06 kernel: Call Trace: Jul 18 07:01:17 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 07:01:17 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 07:01:17 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 07:01:17 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 07:01:17 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Jul 18 07:01:17 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Jul 18 07:01:17 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Jul 18 07:01:17 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Jul 18 07:01:17 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 07:01:17 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 07:01:17 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 07:01:17 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 07:01:17 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 07:01:17 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 07:01:17 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 07:01:17 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 07:01:17 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 07:01:17 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 07:01:17 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 07:01:17 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 07:01:17 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 07:01:17 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 07:01:17 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 07:01:17 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 07:01:17 oak-gw06 kernel: Mem-Info: Jul 18 07:01:17 oak-gw06 kernel: active_anon:30937 inactive_anon:36856 isolated_anon:0#012 active_file:36393 inactive_file:2071395 isolated_file:6#012 unevictable:0 dirty:22619 writeback:7122 unstable:0#012 slab_reclaimable:51926 slab_unreclaimable:894942#012 mapped:10097 shmem:30808 pagetables:1639 bounce:0#012 free:687556 free_pcp:2295 free_cma:0 Jul 18 07:01:17 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 07:01:17 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 07:01:17 oak-gw06 kernel: Node 0 DMA32 free:496068kB min:11976kB low:14968kB high:17964kB active_anon:18848kB inactive_anon:27040kB active_file:21876kB inactive_file:1486344kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:17584kB writeback:6276kB mapped:4668kB shmem:24540kB slab_reclaimable:38168kB slab_unreclaimable:628404kB kernel_stack:976kB pagetables:1316kB unstable:0kB bounce:0kB free_pcp:4440kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 07:01:17 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 07:01:17 oak-gw06 kernel: Node 0 Normal free:2258780kB min:55536kB low:69420kB high:83304kB active_anon:105160kB inactive_anon:120384kB active_file:123696kB inactive_file:6781216kB unevictable:0kB isolated(anon):0kB isolated(file):24kB present:13631488kB managed:13367060kB mlocked:0kB dirty:72888kB writeback:23088kB mapped:35720kB shmem:98692kB slab_reclaimable:169536kB slab_unreclaimable:2951076kB kernel_stack:4704kB pagetables:5240kB unstable:0kB bounce:0kB free_pcp:4312kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 07:01:17 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 07:01:17 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 07:01:17 oak-gw06 kernel: Node 0 DMA32: 1106*4kB (UEM) 1803*8kB (UEM) 4359*16kB (UEM) 1664*32kB (UEM) 1375*64kB (UEM) 1639*128kB (UM) 222*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 496464kB Jul 18 07:01:17 oak-gw06 kernel: Node 0 Normal: 5802*4kB (UEM) 13412*8kB (UEM) 16584*16kB (UEM) 7566*32kB (UEM) 9113*64kB (UEM) 6681*128kB (UEM) 700*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2255560kB Jul 18 07:01:17 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 07:01:17 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 07:01:17 oak-gw06 kernel: 2086008 total pagecache pages Jul 18 07:01:17 oak-gw06 kernel: 0 pages in swap cache Jul 18 07:01:17 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 07:01:17 oak-gw06 kernel: Free swap = 4194300kB Jul 18 07:01:17 oak-gw06 kernel: Total swap = 4194300kB Jul 18 07:01:17 oak-gw06 kernel: 4194203 pages RAM Jul 18 07:01:17 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 07:01:17 oak-gw06 kernel: 127313 pages reserved Jul 18 08:01:17 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Jul 18 08:01:17 oak-gw06 kernel: CPU: 7 PID: 15130 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:01:17 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:01:17 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:01:17 oak-gw06 kernel: 00000000000080d0 0000000064d30854 ffff88007367b858 ffffffff8168662f Jul 18 08:01:17 oak-gw06 kernel: ffff88007367b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Jul 18 08:01:17 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88007367b8b8 0000000064d30854 Jul 18 08:01:17 oak-gw06 kernel: Call Trace: Jul 18 08:01:17 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:01:17 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:01:17 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:01:17 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:01:17 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Jul 18 08:01:17 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Jul 18 08:01:17 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:01:17 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:01:17 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:01:17 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:01:17 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:01:17 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:01:17 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:01:17 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:01:17 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:01:17 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:01:17 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:01:17 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:01:17 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:01:17 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:01:17 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:01:17 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:01:17 oak-gw06 kernel: Mem-Info: Jul 18 08:01:17 oak-gw06 kernel: active_anon:31487 inactive_anon:36856 isolated_anon:0#012 active_file:36405 inactive_file:2302254 isolated_file:0#012 unevictable:0 dirty:18541 writeback:5900 unstable:0#012 slab_reclaimable:51918 slab_unreclaimable:896611#012 mapped:10117 shmem:30808 pagetables:1637 bounce:0#012 free:440728 free_pcp:562 free_cma:0 Jul 18 08:01:17 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:01:17 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:01:17 oak-gw06 kernel: Node 0 DMA32 free:293464kB min:11976kB low:14968kB high:17964kB active_anon:17424kB inactive_anon:27040kB active_file:21884kB inactive_file:1664704kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:12892kB writeback:4300kB mapped:4668kB shmem:24540kB slab_reclaimable:38152kB slab_unreclaimable:630536kB kernel_stack:976kB pagetables:1300kB unstable:0kB bounce:0kB free_pcp:1264kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:01:17 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:01:17 oak-gw06 kernel: Node 0 Normal free:1444944kB min:55536kB low:69420kB high:83304kB active_anon:108524kB inactive_anon:120384kB active_file:123736kB inactive_file:7551940kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:58988kB writeback:20032kB mapped:35800kB shmem:98692kB slab_reclaimable:169520kB slab_unreclaimable:2955892kB kernel_stack:4688kB pagetables:5248kB unstable:0kB bounce:0kB free_pcp:2396kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:01:17 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:01:17 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:01:17 oak-gw06 kernel: Node 0 DMA32: 401*4kB (UEM) 235*8kB (UEM) 217*16kB (UEM) 62*32kB (UEM) 1084*64kB (UEM) 1103*128kB (UM) 284*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 292204kB Jul 18 08:01:17 oak-gw06 kernel: Node 0 Normal: 1588*4kB (UEM) 1344*8kB (UE) 546*16kB (UE) 121*32kB (UE) 6406*64kB (UEM) 5829*128kB (EM) 1001*256kB (EM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1442064kB Jul 18 08:01:17 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:01:17 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:01:17 oak-gw06 kernel: 2104334 total pagecache pages Jul 18 08:01:17 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:01:17 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:01:17 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:01:17 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:01:17 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:01:17 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:01:17 oak-gw06 kernel: 127313 pages reserved Jul 18 08:01:17 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Jul 18 08:01:17 oak-gw06 kernel: CPU: 7 PID: 15130 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:01:17 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:01:17 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:01:17 oak-gw06 kernel: 00000000000080d0 0000000064d30854 ffff88007367b808 ffffffff8168662f Jul 18 08:01:17 oak-gw06 kernel: ffff88007367b898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Jul 18 08:01:17 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88007367b868 0000000064d30854 Jul 18 08:01:17 oak-gw06 kernel: Call Trace: Jul 18 08:01:17 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:01:17 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:01:17 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Jul 18 08:01:17 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:01:17 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:01:17 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Jul 18 08:01:17 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Jul 18 08:01:17 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Jul 18 08:01:17 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Jul 18 08:01:17 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:01:17 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:01:17 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:01:17 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:01:17 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:01:17 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:01:17 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:01:17 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:01:17 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:01:17 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:01:17 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:01:17 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:01:17 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:01:17 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:01:17 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:01:17 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:01:17 oak-gw06 kernel: Mem-Info: Jul 18 08:01:17 oak-gw06 kernel: active_anon:31487 inactive_anon:36856 isolated_anon:0#012 active_file:36405 inactive_file:2305714 isolated_file:0#012 unevictable:0 dirty:18110 writeback:6277 unstable:0#012 slab_reclaimable:51918 slab_unreclaimable:896611#012 mapped:10117 shmem:30808 pagetables:1637 bounce:0#012 free:436930 free_pcp:455 free_cma:0 Jul 18 08:01:17 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:01:17 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:01:17 oak-gw06 kernel: Node 0 DMA32 free:292412kB min:11976kB low:14968kB high:17964kB active_anon:17424kB inactive_anon:27040kB active_file:21884kB inactive_file:1667336kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:14012kB writeback:3740kB mapped:4668kB shmem:24540kB slab_reclaimable:38152kB slab_unreclaimable:630536kB kernel_stack:976kB pagetables:1300kB unstable:0kB bounce:0kB free_pcp:1516kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:01:17 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:01:17 oak-gw06 kernel: Node 0 Normal free:1433860kB min:55536kB low:69420kB high:83304kB active_anon:108524kB inactive_anon:120384kB active_file:123736kB inactive_file:7562600kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:58600kB writeback:20032kB mapped:35800kB shmem:98692kB slab_reclaimable:169520kB slab_unreclaimable:2955892kB kernel_stack:4688kB pagetables:5248kB unstable:0kB bounce:0kB free_pcp:3248kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:01:17 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:01:17 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:01:17 oak-gw06 kernel: Node 0 DMA32: 308*4kB (UEM) 278*8kB (UEM) 254*16kB (UEM) 61*32kB (UE) 1048*64kB (UEM) 1103*128kB (UM) 284*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 290432kB Jul 18 08:01:17 oak-gw06 kernel: Node 0 Normal: 1661*4kB (UEM) 1428*8kB (UEM) 608*16kB (UE) 125*32kB (UE) 6197*64kB (UEM) 5829*128kB (EM) 1001*256kB (EM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1430772kB Jul 18 08:01:17 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:01:17 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:01:17 oak-gw06 kernel: 2104151 total pagecache pages Jul 18 08:01:17 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:01:17 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:01:17 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:01:17 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:01:17 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:01:17 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:01:17 oak-gw06 kernel: 127313 pages reserved Jul 18 08:11:17 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Jul 18 08:11:17 oak-gw06 kernel: CPU: 7 PID: 15183 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:11:17 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:11:17 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:11:17 oak-gw06 kernel: 00000000000080d0 00000000dcc84fdf ffff880210aaf858 ffffffff8168662f Jul 18 08:11:17 oak-gw06 kernel: ffff880210aaf8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Jul 18 08:11:17 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880210aaf8b8 00000000dcc84fdf Jul 18 08:11:17 oak-gw06 kernel: Call Trace: Jul 18 08:11:17 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:11:17 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:11:17 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:11:17 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:11:17 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Jul 18 08:11:17 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Jul 18 08:11:17 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:11:17 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:11:17 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:11:17 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:11:17 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:11:17 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:11:17 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:11:17 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:11:17 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:11:17 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:11:17 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:11:17 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:11:17 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:11:17 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:11:17 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:11:17 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:11:17 oak-gw06 kernel: Mem-Info: Jul 18 08:11:17 oak-gw06 kernel: active_anon:31487 inactive_anon:36856 isolated_anon:0#012 active_file:36406 inactive_file:2235592 isolated_file:0#012 unevictable:0 dirty:10300 writeback:6901 unstable:0#012 slab_reclaimable:51918 slab_unreclaimable:896558#012 mapped:10129 shmem:30808 pagetables:1637 bounce:0#012 free:506136 free_pcp:1382 free_cma:0 Jul 18 08:11:17 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:11:18 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:11:18 oak-gw06 kernel: Node 0 DMA32 free:353020kB min:11976kB low:14968kB high:17964kB active_anon:19616kB inactive_anon:27040kB active_file:21884kB inactive_file:1609364kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:10396kB writeback:5108kB mapped:4668kB shmem:24540kB slab_reclaimable:38152kB slab_unreclaimable:630648kB kernel_stack:992kB pagetables:1296kB unstable:0kB bounce:0kB free_pcp:2600kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:11:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:11:18 oak-gw06 kernel: Node 0 Normal free:1643464kB min:55536kB low:69420kB high:83304kB active_anon:106592kB inactive_anon:120384kB active_file:123740kB inactive_file:7346844kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:38132kB writeback:20384kB mapped:35848kB shmem:98692kB slab_reclaimable:169520kB slab_unreclaimable:2955568kB kernel_stack:4720kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:2496kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:11:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:11:18 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:11:18 oak-gw06 kernel: Node 0 DMA32: 945*4kB (UEM) 853*8kB (UEM) 2250*16kB (UEM) 629*32kB (UEM) 600*64kB (EM) 1387*128kB (UM) 272*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 352300kB Jul 18 08:11:18 oak-gw06 kernel: Node 0 Normal: 1083*4kB (UEM) 7416*8kB (UEM) 8964*16kB (UEM) 3517*32kB (UEM) 1232*64kB (UEM) 7706*128kB (UEM) 976*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1634700kB Jul 18 08:11:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:11:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:11:18 oak-gw06 kernel: 2104316 total pagecache pages Jul 18 08:11:18 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:11:18 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:11:18 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:11:18 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:11:18 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:11:18 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:11:18 oak-gw06 kernel: 127313 pages reserved Jul 18 08:11:18 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Jul 18 08:11:18 oak-gw06 kernel: CPU: 7 PID: 15183 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:11:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:11:18 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:11:18 oak-gw06 kernel: 00000000000080d0 00000000dcc84fdf ffff880210aaf808 ffffffff8168662f Jul 18 08:11:18 oak-gw06 kernel: ffff880210aaf898 ffffffff81186ba0 0000000000000000 00000000ffffffff Jul 18 08:11:18 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880210aaf868 00000000dcc84fdf Jul 18 08:11:18 oak-gw06 kernel: Call Trace: Jul 18 08:11:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:11:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:11:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:11:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:11:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Jul 18 08:11:18 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Jul 18 08:11:18 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Jul 18 08:11:18 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Jul 18 08:11:18 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:11:18 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:11:18 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:11:18 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:11:18 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:11:18 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:11:18 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:11:18 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:11:18 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:11:18 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:11:18 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:11:18 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:11:18 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:11:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:11:18 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:11:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:11:18 oak-gw06 kernel: Mem-Info: Jul 18 08:11:18 oak-gw06 kernel: active_anon:31487 inactive_anon:36856 isolated_anon:0#012 active_file:36406 inactive_file:2247037 isolated_file:0#012 unevictable:0 dirty:13550 writeback:6103 unstable:0#012 slab_reclaimable:51918 slab_unreclaimable:896558#012 mapped:10129 shmem:30808 pagetables:1637 bounce:0#012 free:495034 free_pcp:888 free_cma:0 Jul 18 08:11:18 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:11:18 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:11:18 oak-gw06 kernel: Node 0 DMA32 free:349088kB min:11976kB low:14968kB high:17964kB active_anon:19616kB inactive_anon:27040kB active_file:21884kB inactive_file:1618128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:12192kB writeback:5968kB mapped:4668kB shmem:24540kB slab_reclaimable:38152kB slab_unreclaimable:630648kB kernel_stack:992kB pagetables:1296kB unstable:0kB bounce:0kB free_pcp:1740kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:11:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:11:18 oak-gw06 kernel: Node 0 Normal free:1607724kB min:55536kB low:69420kB high:83304kB active_anon:106332kB inactive_anon:120384kB active_file:123740kB inactive_file:7379228kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:35288kB writeback:20024kB mapped:35856kB shmem:98692kB slab_reclaimable:169520kB slab_unreclaimable:2955888kB kernel_stack:4720kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:2452kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:11:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:11:18 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:11:18 oak-gw06 kernel: Node 0 DMA32: 1381*4kB (UE) 785*8kB (UE) 1780*16kB (UEM) 653*32kB (UEM) 600*64kB (EM) 1387*128kB (UM) 272*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 346748kB Jul 18 08:11:18 oak-gw06 kernel: Node 0 Normal: 1242*4kB (UEM) 3454*8kB (UEM) 9012*16kB (UEM) 3502*32kB (UEM) 1220*64kB (UEM) 7704*128kB (UEM) 973*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1602136kB Jul 18 08:11:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:11:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:11:18 oak-gw06 kernel: 2104207 total pagecache pages Jul 18 08:11:18 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:11:18 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:11:18 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:11:18 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:11:18 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:11:18 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:11:18 oak-gw06 kernel: 127313 pages reserved Jul 18 08:16:17 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Jul 18 08:16:17 oak-gw06 kernel: CPU: 3 PID: 15193 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:16:17 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:16:17 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:16:17 oak-gw06 kernel: 00000000000080d0 0000000025516c94 ffff8801ac74f858 ffffffff8168662f Jul 18 08:16:17 oak-gw06 kernel: ffff8801ac74f8e8 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Jul 18 08:16:17 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 0000000025516c94 Jul 18 08:16:17 oak-gw06 kernel: Call Trace: Jul 18 08:16:17 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:16:17 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:16:17 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Jul 18 08:16:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:16:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:16:18 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Jul 18 08:16:18 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Jul 18 08:16:18 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:16:18 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:16:18 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:16:18 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:16:18 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:16:18 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:16:18 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:16:18 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:16:18 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:16:18 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:16:18 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:16:18 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:16:18 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:16:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:16:18 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:16:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:16:18 oak-gw06 kernel: Mem-Info: Jul 18 08:16:18 oak-gw06 kernel: active_anon:31488 inactive_anon:36856 isolated_anon:0#012 active_file:36406 inactive_file:2234951 isolated_file:0#012 unevictable:0 dirty:11307 writeback:6168 unstable:0#012 slab_reclaimable:51918 slab_unreclaimable:896210#012 mapped:10142 shmem:30808 pagetables:1637 bounce:0#012 free:508865 free_pcp:876 free_cma:0 Jul 18 08:16:18 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:16:18 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:16:18 oak-gw06 kernel: Node 0 DMA32 free:354248kB min:11976kB low:14968kB high:17964kB active_anon:19620kB inactive_anon:27040kB active_file:21884kB inactive_file:1625936kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:8916kB writeback:3748kB mapped:4668kB shmem:24540kB slab_reclaimable:38152kB slab_unreclaimable:629760kB kernel_stack:960kB pagetables:1296kB unstable:0kB bounce:0kB free_pcp:1500kB local_pcp:64kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:16:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:16:18 oak-gw06 kernel: Node 0 Normal free:1691384kB min:55536kB low:69420kB high:83304kB active_anon:106332kB inactive_anon:120384kB active_file:123740kB inactive_file:7286252kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:41096kB writeback:19932kB mapped:35900kB shmem:98692kB slab_reclaimable:169520kB slab_unreclaimable:2955064kB kernel_stack:4704kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:3072kB local_pcp:84kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:16:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:16:18 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:16:18 oak-gw06 kernel: Node 0 DMA32: 2713*4kB (UEM) 990*8kB (UEM) 1208*16kB (UEM) 313*32kB (UEM) 637*64kB (UEM) 1618*128kB (UM) 287*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 369460kB Jul 18 08:16:18 oak-gw06 kernel: Node 0 Normal: 13803*4kB (UEM) 6047*8kB (UEM) 2162*16kB (UEM) 105*32kB (UEM) 6021*64kB (EM) 7857*128kB (UEM) 942*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1773732kB Jul 18 08:16:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:16:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:16:18 oak-gw06 kernel: 2101289 total pagecache pages Jul 18 08:16:18 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:16:18 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:16:18 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:16:18 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:16:18 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:16:18 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:16:18 oak-gw06 kernel: 127313 pages reserved Jul 18 08:16:18 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Jul 18 08:16:18 oak-gw06 kernel: CPU: 3 PID: 15193 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:16:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:16:18 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:16:18 oak-gw06 kernel: 00000000000080d0 0000000025516c94 ffff8801ac74f808 ffffffff8168662f Jul 18 08:16:18 oak-gw06 kernel: ffff8801ac74f898 ffffffff81186ba0 0000000000000000 00000000ffffffff Jul 18 08:16:18 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801ac74f868 0000000025516c94 Jul 18 08:16:18 oak-gw06 kernel: Call Trace: Jul 18 08:16:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:16:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:16:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:16:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:16:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Jul 18 08:16:18 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Jul 18 08:16:18 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Jul 18 08:16:18 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Jul 18 08:16:18 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:16:18 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:16:18 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:16:18 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:16:18 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:16:18 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:16:18 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:16:18 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:16:18 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:16:18 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:16:18 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:16:18 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:16:18 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:16:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:16:18 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:16:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:16:18 oak-gw06 kernel: Mem-Info: Jul 18 08:16:18 oak-gw06 kernel: active_anon:31488 inactive_anon:36856 isolated_anon:0#012 active_file:36406 inactive_file:2121781 isolated_file:0#012 unevictable:0 dirty:16232 writeback:6135 unstable:0#012 slab_reclaimable:51918 slab_unreclaimable:896534#012 mapped:10142 shmem:30808 pagetables:1637 bounce:0#012 free:621582 free_pcp:891 free_cma:0 Jul 18 08:16:18 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:16:18 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:16:18 oak-gw06 kernel: Node 0 DMA32 free:449016kB min:11976kB low:14968kB high:17964kB active_anon:19620kB inactive_anon:27040kB active_file:21884kB inactive_file:1531488kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:9768kB writeback:2792kB mapped:4668kB shmem:24540kB slab_reclaimable:38152kB slab_unreclaimable:629840kB kernel_stack:960kB pagetables:1296kB unstable:0kB bounce:0kB free_pcp:1572kB local_pcp:72kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:16:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:16:18 oak-gw06 kernel: Node 0 Normal free:2064284kB min:55536kB low:69420kB high:83304kB active_anon:106332kB inactive_anon:120384kB active_file:123740kB inactive_file:6912620kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:40208kB writeback:20276kB mapped:35944kB shmem:98692kB slab_reclaimable:169520kB slab_unreclaimable:2956152kB kernel_stack:4720kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:4088kB local_pcp:8kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:16:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:16:18 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:16:18 oak-gw06 kernel: Node 0 DMA32: 3178*4kB (UEM) 4408*8kB (UEM) 3231*16kB (UEM) 805*32kB (UEM) 664*64kB (UEM) 1618*128kB (UM) 287*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 448504kB Jul 18 08:16:18 oak-gw06 kernel: Node 0 Normal: 16664*4kB (UEM) 19058*8kB (UEM) 10086*16kB (UEM) 1819*32kB (UEM) 6043*64kB (EM) 7857*128kB (UEM) 942*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2072304kB Jul 18 08:16:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:16:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:16:18 oak-gw06 kernel: 2073278 total pagecache pages Jul 18 08:16:18 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:16:18 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:16:18 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:16:18 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:16:18 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:16:18 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:16:18 oak-gw06 kernel: 127313 pages reserved Jul 18 08:21:18 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Jul 18 08:21:18 oak-gw06 kernel: CPU: 7 PID: 15193 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:21:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:21:18 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:21:18 oak-gw06 kernel: 00000000000080d0 0000000025516c94 ffff8801ac74f858 ffffffff8168662f Jul 18 08:21:18 oak-gw06 kernel: ffff8801ac74f8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Jul 18 08:21:18 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801ac74f8b8 0000000025516c94 Jul 18 08:21:18 oak-gw06 kernel: Call Trace: Jul 18 08:21:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:21:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:21:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:21:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:21:18 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Jul 18 08:21:18 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Jul 18 08:21:18 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:21:18 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:21:18 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:21:18 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:21:18 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:21:18 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:21:18 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:21:18 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:21:18 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:21:18 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:21:18 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:21:18 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:21:18 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:21:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:21:18 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:21:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:21:18 oak-gw06 kernel: Mem-Info: Jul 18 08:21:18 oak-gw06 kernel: active_anon:31488 inactive_anon:36856 isolated_anon:0#012 active_file:36407 inactive_file:2227135 isolated_file:0#012 unevictable:0 dirty:19698 writeback:8000 unstable:0#012 slab_reclaimable:51894 slab_unreclaimable:896757#012 mapped:10155 shmem:30808 pagetables:1637 bounce:0#012 free:516447 free_pcp:790 free_cma:0 Jul 18 08:21:18 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:21:18 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:21:18 oak-gw06 kernel: Node 0 DMA32 free:361732kB min:11976kB low:14968kB high:17964kB active_anon:20888kB inactive_anon:27040kB active_file:21884kB inactive_file:1607632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:12816kB writeback:4864kB mapped:4668kB shmem:24540kB slab_reclaimable:38148kB slab_unreclaimable:630632kB kernel_stack:976kB pagetables:1296kB unstable:0kB bounce:0kB free_pcp:1584kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:21:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:21:18 oak-gw06 kernel: Node 0 Normal free:1681296kB min:55536kB low:69420kB high:83304kB active_anon:105064kB inactive_anon:120384kB active_file:123744kB inactive_file:7306368kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:68304kB writeback:27352kB mapped:35952kB shmem:98692kB slab_reclaimable:169428kB slab_unreclaimable:2956380kB kernel_stack:4720kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:2384kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:21:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:21:18 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:21:18 oak-gw06 kernel: Node 0 DMA32: 838*4kB (UEM) 473*8kB (UEM) 1128*16kB (UEM) 1518*32kB (UEM) 590*64kB (UEM) 1371*128kB (UM) 283*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 359968kB Jul 18 08:21:18 oak-gw06 kernel: Node 0 Normal: 1047*4kB (UEM) 566*8kB (UE) 2660*16kB (UEM) 6751*32kB (UEM) 4150*64kB (EM) 7197*128kB (UEM) 865*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1675564kB Jul 18 08:21:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:21:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:21:18 oak-gw06 kernel: 2103929 total pagecache pages Jul 18 08:21:18 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:21:18 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:21:18 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:21:18 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:21:18 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:21:18 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:21:18 oak-gw06 kernel: 127313 pages reserved Jul 18 08:31:18 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Jul 18 08:31:18 oak-gw06 kernel: CPU: 3 PID: 15195 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:31:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:31:18 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:31:18 oak-gw06 kernel: 00000000000080d0 000000009ca056d4 ffff88012b91f858 ffffffff8168662f Jul 18 08:31:18 oak-gw06 kernel: ffff88012b91f8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Jul 18 08:31:18 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88012b91f8b8 000000009ca056d4 Jul 18 08:31:18 oak-gw06 kernel: Call Trace: Jul 18 08:31:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:31:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:31:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:31:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:31:18 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Jul 18 08:31:18 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Jul 18 08:31:18 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:31:18 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:31:18 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:31:18 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:31:18 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:31:18 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:31:18 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:31:18 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:31:18 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:31:18 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:31:18 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:31:18 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:31:18 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:31:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:31:18 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:31:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:31:18 oak-gw06 kernel: Mem-Info: Jul 18 08:31:18 oak-gw06 kernel: active_anon:31488 inactive_anon:36856 isolated_anon:0#012 active_file:36409 inactive_file:2247475 isolated_file:0#012 unevictable:0 dirty:19046 writeback:7668 unstable:0#012 slab_reclaimable:51894 slab_unreclaimable:896363#012 mapped:10162 shmem:30808 pagetables:1637 bounce:0#012 free:480277 free_pcp:1353 free_cma:0 Jul 18 08:31:18 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:31:18 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:31:18 oak-gw06 kernel: Node 0 DMA32 free:327356kB min:11976kB low:14968kB high:17964kB active_anon:19836kB inactive_anon:27040kB active_file:21884kB inactive_file:1627808kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:14220kB writeback:5868kB mapped:4668kB shmem:24540kB slab_reclaimable:38148kB slab_unreclaimable:630304kB kernel_stack:976kB pagetables:1300kB unstable:0kB bounce:0kB free_pcp:2928kB local_pcp:100kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:31:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:31:18 oak-gw06 kernel: Node 0 Normal free:1575268kB min:55536kB low:69420kB high:83304kB active_anon:106116kB inactive_anon:120384kB active_file:123752kB inactive_file:7364432kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:64680kB writeback:24804kB mapped:35980kB shmem:98692kB slab_reclaimable:169428kB slab_unreclaimable:2955132kB kernel_stack:4704kB pagetables:5248kB unstable:0kB bounce:0kB free_pcp:3204kB local_pcp:84kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:31:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:31:18 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:31:18 oak-gw06 kernel: Node 0 DMA32: 1090*4kB (UE) 297*8kB (UE) 366*16kB (UE) 113*32kB (UEM) 894*64kB (UEM) 1531*128kB (UM) 232*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 328784kB Jul 18 08:31:18 oak-gw06 kernel: Node 0 Normal: 1022*4kB (UE) 407*8kB (UEM) 542*16kB (UEM) 1379*32kB (EM) 6197*64kB (EM) 7430*128kB (EM) 644*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1572656kB Jul 18 08:31:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:31:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:31:18 oak-gw06 kernel: 2104735 total pagecache pages Jul 18 08:31:18 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:31:18 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:31:18 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:31:18 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:31:18 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:31:18 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:31:18 oak-gw06 kernel: 127313 pages reserved Jul 18 08:31:18 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Jul 18 08:31:18 oak-gw06 kernel: CPU: 2 PID: 15195 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:31:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:31:18 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:31:18 oak-gw06 kernel: 00000000000080d0 000000009ca056d4 ffff88012b91f808 ffffffff8168662f Jul 18 08:31:18 oak-gw06 kernel: ffff88012b91f898 ffffffff81186ba0 0000000000000000 00000000ffffffff Jul 18 08:31:18 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88012b91f868 000000009ca056d4 Jul 18 08:31:18 oak-gw06 kernel: Call Trace: Jul 18 08:31:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:31:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:31:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:31:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:31:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Jul 18 08:31:18 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Jul 18 08:31:18 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Jul 18 08:31:18 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Jul 18 08:31:18 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:31:18 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:31:18 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:31:18 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:31:18 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:31:18 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:31:18 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:31:18 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:31:18 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:31:18 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:31:18 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:31:18 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:31:18 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:31:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:31:18 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:31:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:31:18 oak-gw06 kernel: Mem-Info: Jul 18 08:31:18 oak-gw06 kernel: active_anon:31488 inactive_anon:36856 isolated_anon:0#012 active_file:36409 inactive_file:2250385 isolated_file:0#012 unevictable:0 dirty:22279 writeback:7668 unstable:0#012 slab_reclaimable:51894 slab_unreclaimable:896363#012 mapped:10162 shmem:30808 pagetables:1637 bounce:0#012 free:476902 free_pcp:1251 free_cma:0 Jul 18 08:31:18 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:31:18 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:31:18 oak-gw06 kernel: Node 0 DMA32 free:326440kB min:11976kB low:14968kB high:17964kB active_anon:19836kB inactive_anon:27040kB active_file:21884kB inactive_file:1629688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:17580kB writeback:5868kB mapped:4668kB shmem:24540kB slab_reclaimable:38148kB slab_unreclaimable:630304kB kernel_stack:976kB pagetables:1300kB unstable:0kB bounce:0kB free_pcp:2192kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:31:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:31:18 oak-gw06 kernel: Node 0 Normal free:1563080kB min:55536kB low:69420kB high:83304kB active_anon:106636kB inactive_anon:120384kB active_file:123752kB inactive_file:7376132kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:75932kB writeback:24804kB mapped:35980kB shmem:98692kB slab_reclaimable:169428kB slab_unreclaimable:2955132kB kernel_stack:4704kB pagetables:5248kB unstable:0kB bounce:0kB free_pcp:4328kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:31:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:31:18 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:31:18 oak-gw06 kernel: Node 0 DMA32: 1023*4kB (UEM) 349*8kB (UE) 412*16kB (UEM) 51*32kB (UE) 891*64kB (UEM) 1531*128kB (UM) 232*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 327492kB Jul 18 08:31:18 oak-gw06 kernel: Node 0 Normal: 1256*4kB (UE) 441*8kB (UE) 538*16kB (UEM) 945*32kB (EM) 6197*64kB (UEM) 7426*128kB (EM) 640*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1558376kB Jul 18 08:31:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:31:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:31:18 oak-gw06 kernel: 2104719 total pagecache pages Jul 18 08:31:18 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:31:18 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:31:18 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:31:18 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:31:18 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:31:18 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:31:18 oak-gw06 kernel: 127313 pages reserved Jul 18 08:36:19 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Jul 18 08:36:19 oak-gw06 kernel: CPU: 0 PID: 15195 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:36:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:36:19 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:36:19 oak-gw06 kernel: 00000000000080d0 000000009ca056d4 ffff88012b91f858 ffffffff8168662f Jul 18 08:36:19 oak-gw06 kernel: ffff88012b91f8e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Jul 18 08:36:19 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff88012b91f8e8 000000009ca056d4 Jul 18 08:36:19 oak-gw06 kernel: Call Trace: Jul 18 08:36:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:36:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:36:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:36:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:36:19 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Jul 18 08:36:19 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Jul 18 08:36:19 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:36:19 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:36:19 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:36:19 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:36:19 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:36:19 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:36:19 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:36:19 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:36:19 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:36:19 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:36:19 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:36:19 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:36:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:36:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:36:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:36:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:36:19 oak-gw06 kernel: Mem-Info: Jul 18 08:36:19 oak-gw06 kernel: active_anon:31496 inactive_anon:36856 isolated_anon:0#012 active_file:36409 inactive_file:2269458 isolated_file:12#012 unevictable:0 dirty:22657 writeback:7254 unstable:0#012 slab_reclaimable:51894 slab_unreclaimable:896507#012 mapped:10171 shmem:30808 pagetables:1637 bounce:0#012 free:459472 free_pcp:1332 free_cma:0 Jul 18 08:36:19 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:36:19 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:36:19 oak-gw06 kernel: Node 0 DMA32 free:307068kB min:11976kB low:14968kB high:17964kB active_anon:19860kB inactive_anon:27040kB active_file:21884kB inactive_file:1635788kB unevictable:0kB isolated(anon):0kB isolated(file):48kB present:3129332kB managed:2884592kB mlocked:0kB dirty:17520kB writeback:5076kB mapped:4668kB shmem:24540kB slab_reclaimable:38148kB slab_unreclaimable:630208kB kernel_stack:976kB pagetables:1300kB unstable:0kB bounce:0kB free_pcp:2896kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:36:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:36:19 oak-gw06 kernel: Node 0 Normal free:1510492kB min:55536kB low:69420kB high:83304kB active_anon:106124kB inactive_anon:120384kB active_file:123752kB inactive_file:7445036kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:66316kB writeback:21128kB mapped:36016kB shmem:98692kB slab_reclaimable:169428kB slab_unreclaimable:2955804kB kernel_stack:4720kB pagetables:5248kB unstable:0kB bounce:0kB free_pcp:2576kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:36:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:36:19 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:36:19 oak-gw06 kernel: Node 0 DMA32: 433*4kB (UEM) 233*8kB (UE) 156*16kB (UE) 803*32kB (EM) 908*64kB (UEM) 1273*128kB (UM) 215*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 307884kB Jul 18 08:36:19 oak-gw06 kernel: Node 0 Normal: 1309*4kB (UEM) 1211*8kB (UEM) 800*16kB (UEM) 4536*32kB (UEM) 4982*64kB (UEM) 6689*128kB (UEM) 625*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1507916kB Jul 18 08:36:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:36:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:36:19 oak-gw06 kernel: 2104541 total pagecache pages Jul 18 08:36:19 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:36:19 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:36:19 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:36:19 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:36:19 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:36:19 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:36:19 oak-gw06 kernel: 127313 pages reserved Jul 18 08:36:19 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Jul 18 08:36:19 oak-gw06 kernel: CPU: 0 PID: 15195 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:36:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:36:19 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:36:19 oak-gw06 kernel: 00000000000080d0 000000009ca056d4 ffff88012b91f808 ffffffff8168662f Jul 18 08:36:19 oak-gw06 kernel: ffff88012b91f898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Jul 18 08:36:19 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88012b91f868 000000009ca056d4 Jul 18 08:36:19 oak-gw06 kernel: Call Trace: Jul 18 08:36:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:36:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:36:19 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Jul 18 08:36:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:36:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:36:19 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Jul 18 08:36:19 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Jul 18 08:36:19 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Jul 18 08:36:19 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Jul 18 08:36:19 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:36:19 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:36:19 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:36:19 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:36:19 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:36:19 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:36:19 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:36:19 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:36:19 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:36:19 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:36:19 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:36:19 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:36:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:36:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:36:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:36:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:36:19 oak-gw06 kernel: Mem-Info: Jul 18 08:36:19 oak-gw06 kernel: active_anon:31496 inactive_anon:36856 isolated_anon:0#012 active_file:36409 inactive_file:2272723 isolated_file:12#012 unevictable:0 dirty:22045 writeback:6890 unstable:0#012 slab_reclaimable:51894 slab_unreclaimable:896507#012 mapped:10171 shmem:30808 pagetables:1637 bounce:0#012 free:456221 free_pcp:494 free_cma:0 Jul 18 08:36:19 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:36:19 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:36:19 oak-gw06 kernel: Node 0 DMA32 free:306732kB min:11976kB low:14968kB high:17964kB active_anon:19860kB inactive_anon:27040kB active_file:21884kB inactive_file:1639040kB unevictable:0kB isolated(anon):0kB isolated(file):48kB present:3129332kB managed:2884592kB mlocked:0kB dirty:19492kB writeback:4492kB mapped:4668kB shmem:24540kB slab_reclaimable:38148kB slab_unreclaimable:630208kB kernel_stack:976kB pagetables:1300kB unstable:0kB bounce:0kB free_pcp:1020kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:36:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:36:19 oak-gw06 kernel: Node 0 Normal free:1495596kB min:55536kB low:69420kB high:83304kB active_anon:106384kB inactive_anon:120384kB active_file:123752kB inactive_file:7457256kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:71748kB writeback:23844kB mapped:36016kB shmem:98692kB slab_reclaimable:169428kB slab_unreclaimable:2955804kB kernel_stack:4720kB pagetables:5248kB unstable:0kB bounce:0kB free_pcp:1252kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:36:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:36:19 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:36:19 oak-gw06 kernel: Node 0 DMA32: 740*4kB (UEM) 298*8kB (UE) 167*16kB (UE) 693*32kB (UEM) 907*64kB (UEM) 1272*128kB (M) 215*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 306096kB Jul 18 08:36:19 oak-gw06 kernel: Node 0 Normal: 1192*4kB (UE) 1204*8kB (UE) 756*16kB (UEM) 4136*32kB (UEM) 4982*64kB (UEM) 6689*128kB (UEM) 625*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1493888kB Jul 18 08:36:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:36:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:36:19 oak-gw06 kernel: 2104293 total pagecache pages Jul 18 08:36:19 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:36:19 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:36:19 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:36:19 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:36:19 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:36:19 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:36:19 oak-gw06 kernel: 127313 pages reserved Jul 18 08:41:19 oak-gw06 kernel: kworker/u16:3: page allocation failure: order:7, mode:0x80d0 Jul 18 08:41:19 oak-gw06 kernel: CPU: 5 PID: 15217 Comm: kworker/u16:3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:41:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:41:19 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:41:19 oak-gw06 kernel: 00000000000080d0 000000003a77e088 ffff880185c9b858 ffffffff8168662f Jul 18 08:41:19 oak-gw06 kernel: ffff880185c9b8e8 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Jul 18 08:41:19 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 000000003a77e088 Jul 18 08:41:19 oak-gw06 kernel: Call Trace: Jul 18 08:41:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:41:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:41:19 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Jul 18 08:41:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:41:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:41:19 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Jul 18 08:41:19 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Jul 18 08:41:19 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:41:19 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:41:19 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:41:19 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:41:19 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:41:19 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:41:19 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:41:19 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:41:19 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:41:19 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:41:19 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:41:19 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:41:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:41:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:41:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:41:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:41:19 oak-gw06 kernel: Mem-Info: Jul 18 08:41:19 oak-gw06 kernel: active_anon:31690 inactive_anon:36856 isolated_anon:0#012 active_file:36410 inactive_file:2147966 isolated_file:0#012 unevictable:0 dirty:24370 writeback:7825 unstable:0#012 slab_reclaimable:51894 slab_unreclaimable:896433#012 mapped:10363 shmem:30808 pagetables:1639 bounce:0#012 free:582770 free_pcp:1178 free_cma:0 Jul 18 08:41:19 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:41:19 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:41:19 oak-gw06 kernel: Node 0 DMA32 free:394884kB min:11976kB low:14968kB high:17964kB active_anon:19916kB inactive_anon:27040kB active_file:21884kB inactive_file:1552456kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:16748kB writeback:7540kB mapped:4668kB shmem:24540kB slab_reclaimable:38148kB slab_unreclaimable:630280kB kernel_stack:976kB pagetables:1304kB unstable:0kB bounce:0kB free_pcp:2296kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:41:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:41:19 oak-gw06 kernel: Node 0 Normal free:1916116kB min:55536kB low:69420kB high:83304kB active_anon:106844kB inactive_anon:120384kB active_file:123756kB inactive_file:7041648kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:83120kB writeback:23760kB mapped:36784kB shmem:98692kB slab_reclaimable:169428kB slab_unreclaimable:2955436kB kernel_stack:4736kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:2488kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:41:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:41:19 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:41:19 oak-gw06 kernel: Node 0 DMA32: 4397*4kB (UEM) 3181*8kB (UEM) 2025*16kB (UEM) 740*32kB (UEM) 759*64kB (EM) 1491*128kB (UM) 215*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 393580kB Jul 18 08:41:19 oak-gw06 kernel: Node 0 Normal: 19047*4kB (UEM) 13739*8kB (UEM) 13470*16kB (UEM) 3444*32kB (UEM) 4245*64kB (UEM) 7345*128kB (UEM) 751*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1915924kB Jul 18 08:41:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:41:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:41:19 oak-gw06 kernel: 2104354 total pagecache pages Jul 18 08:41:19 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:41:19 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:41:19 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:41:19 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:41:19 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:41:19 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:41:19 oak-gw06 kernel: 127313 pages reserved Jul 18 08:41:19 oak-gw06 kernel: kworker/u16:3: page allocation failure: order:7, mode:0x80d0 Jul 18 08:41:19 oak-gw06 kernel: CPU: 5 PID: 15217 Comm: kworker/u16:3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:41:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:41:19 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:41:19 oak-gw06 kernel: 00000000000080d0 000000003a77e088 ffff880185c9b808 ffffffff8168662f Jul 18 08:41:19 oak-gw06 kernel: ffff880185c9b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Jul 18 08:41:19 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880185c9b868 000000003a77e088 Jul 18 08:41:19 oak-gw06 kernel: Call Trace: Jul 18 08:41:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:41:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:41:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:41:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:41:19 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Jul 18 08:41:19 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Jul 18 08:41:19 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Jul 18 08:41:19 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Jul 18 08:41:19 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:41:19 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:41:19 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:41:19 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:41:19 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:41:19 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:41:19 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:41:19 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:41:19 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:41:19 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:41:19 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:41:19 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:41:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:41:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:41:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:41:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:41:19 oak-gw06 kernel: Mem-Info: Jul 18 08:41:19 oak-gw06 kernel: active_anon:31690 inactive_anon:36856 isolated_anon:0#012 active_file:36410 inactive_file:2148432 isolated_file:0#012 unevictable:0 dirty:24967 writeback:7825 unstable:0#012 slab_reclaimable:51894 slab_unreclaimable:896433#012 mapped:10363 shmem:30808 pagetables:1639 bounce:0#012 free:581432 free_pcp:78 free_cma:0 Jul 18 08:41:19 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:41:19 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:41:19 oak-gw06 kernel: Node 0 DMA32 free:394532kB min:11976kB low:14968kB high:17964kB active_anon:19916kB inactive_anon:27040kB active_file:21884kB inactive_file:1552080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:16748kB writeback:7540kB mapped:4668kB shmem:24540kB slab_reclaimable:38148kB slab_unreclaimable:630280kB kernel_stack:976kB pagetables:1304kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:41:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:41:19 oak-gw06 kernel: Node 0 Normal free:1913756kB min:55536kB low:69420kB high:83304kB active_anon:106844kB inactive_anon:120384kB active_file:123756kB inactive_file:7041648kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:83120kB writeback:23760kB mapped:36784kB shmem:98692kB slab_reclaimable:169428kB slab_unreclaimable:2955436kB kernel_stack:4736kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:804kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:41:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:41:19 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:41:19 oak-gw06 kernel: Node 0 DMA32: 4795*4kB (UEM) 3266*8kB (UEM) 2031*16kB (UEM) 743*32kB (UEM) 760*64kB (EM) 1491*128kB (UM) 215*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 396108kB Jul 18 08:41:19 oak-gw06 kernel: Node 0 Normal: 18964*4kB (UEM) 13756*8kB (UEM) 13357*16kB (UEM) 3444*32kB (UEM) 4245*64kB (UEM) 7345*128kB (UEM) 751*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1913920kB Jul 18 08:41:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:41:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:41:19 oak-gw06 kernel: 2104214 total pagecache pages Jul 18 08:41:19 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:41:19 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:41:19 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:41:19 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:41:19 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:41:19 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:41:19 oak-gw06 kernel: 127313 pages reserved Jul 18 08:46:19 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Jul 18 08:46:19 oak-gw06 kernel: CPU: 7 PID: 15195 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:46:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:46:19 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:46:19 oak-gw06 kernel: 00000000000080d0 000000009ca056d4 ffff88012b91f858 ffffffff8168662f Jul 18 08:46:19 oak-gw06 kernel: ffff88012b91f8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Jul 18 08:46:19 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88012b91f8b8 000000009ca056d4 Jul 18 08:46:19 oak-gw06 kernel: Call Trace: Jul 18 08:46:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:46:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:46:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:46:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:46:19 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Jul 18 08:46:19 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Jul 18 08:46:19 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:46:19 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:46:19 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:46:19 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:46:19 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:46:19 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:46:19 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:46:19 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:46:19 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:46:19 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:46:19 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:46:19 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:46:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:46:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:46:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:46:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:46:19 oak-gw06 kernel: Mem-Info: Jul 18 08:46:19 oak-gw06 kernel: active_anon:31710 inactive_anon:36856 isolated_anon:0#012 active_file:36410 inactive_file:1933532 isolated_file:0#012 unevictable:0 dirty:25445 writeback:7480 unstable:0#012 slab_reclaimable:51894 slab_unreclaimable:882583#012 mapped:10203 shmem:30808 pagetables:1638 bounce:0#012 free:810266 free_pcp:627 free_cma:0 Jul 18 08:46:19 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:46:19 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:46:19 oak-gw06 kernel: Node 0 DMA32 free:568364kB min:11976kB low:14968kB high:17964kB active_anon:19936kB inactive_anon:27040kB active_file:21884kB inactive_file:1398704kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:17536kB writeback:2724kB mapped:4668kB shmem:24540kB slab_reclaimable:38148kB slab_unreclaimable:620104kB kernel_stack:976kB pagetables:1304kB unstable:0kB bounce:0kB free_pcp:1432kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:46:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:46:19 oak-gw06 kernel: Node 0 Normal free:2655060kB min:55536kB low:69420kB high:83304kB active_anon:106904kB inactive_anon:120384kB active_file:123756kB inactive_file:6336724kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:82304kB writeback:26808kB mapped:36144kB shmem:98692kB slab_reclaimable:169428kB slab_unreclaimable:2910484kB kernel_stack:4704kB pagetables:5248kB unstable:0kB bounce:0kB free_pcp:2220kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:46:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:46:19 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:46:19 oak-gw06 kernel: Node 0 DMA32: 8377*4kB (UEM) 9587*8kB (UEM) 948*16kB (UEM) 2847*32kB (UEM) 1353*64kB (UEM) 1563*128kB (UM) 237*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 563804kB Jul 18 08:46:19 oak-gw06 kernel: Node 0 Normal: 35702*4kB (UEM) 37397*8kB (UEM) 5004*16kB (UEM) 13436*32kB (UEM) 8511*64kB (UEM) 7377*128kB (UEM) 809*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2648064kB Jul 18 08:46:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:46:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:46:19 oak-gw06 kernel: 2003463 total pagecache pages Jul 18 08:46:19 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:46:19 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:46:19 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:46:19 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:46:19 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:46:19 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:46:19 oak-gw06 kernel: 127313 pages reserved Jul 18 08:46:19 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Jul 18 08:46:19 oak-gw06 kernel: CPU: 7 PID: 15195 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Jul 18 08:46:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Jul 18 08:46:19 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Jul 18 08:46:19 oak-gw06 kernel: 00000000000080d0 000000009ca056d4 ffff88012b91f808 ffffffff8168662f Jul 18 08:46:19 oak-gw06 kernel: ffff88012b91f898 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Jul 18 08:46:19 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 000000009ca056d4 Jul 18 08:46:19 oak-gw06 kernel: Call Trace: Jul 18 08:46:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Jul 18 08:46:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Jul 18 08:46:19 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Jul 18 08:46:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Jul 18 08:46:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Jul 18 08:46:19 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Jul 18 08:46:19 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Jul 18 08:46:19 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Jul 18 08:46:19 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Jul 18 08:46:19 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Jul 18 08:46:19 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Jul 18 08:46:19 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Jul 18 08:46:19 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Jul 18 08:46:19 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Jul 18 08:46:19 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Jul 18 08:46:19 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Jul 18 08:46:19 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Jul 18 08:46:19 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Jul 18 08:46:19 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Jul 18 08:46:19 oak-gw06 kernel: [] worker_thread+0x126/0x410 Jul 18 08:46:19 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Jul 18 08:46:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Jul 18 08:46:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:46:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Jul 18 08:46:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Jul 18 08:46:19 oak-gw06 kernel: Mem-Info: Jul 18 08:46:19 oak-gw06 kernel: active_anon:31710 inactive_anon:36856 isolated_anon:0#012 active_file:36410 inactive_file:1937375 isolated_file:0#012 unevictable:0 dirty:25865 writeback:7049 unstable:0#012 slab_reclaimable:51894 slab_unreclaimable:882991#012 mapped:10203 shmem:30808 pagetables:1638 bounce:0#012 free:804634 free_pcp:344 free_cma:0 Jul 18 08:46:19 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 18 08:46:19 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Jul 18 08:46:19 oak-gw06 kernel: Node 0 DMA32 free:562684kB min:11976kB low:14968kB high:17964kB active_anon:19936kB inactive_anon:27040kB active_file:21884kB inactive_file:1401336kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:19216kB writeback:2164kB mapped:4668kB shmem:24540kB slab_reclaimable:38148kB slab_unreclaimable:620104kB kernel_stack:976kB pagetables:1304kB unstable:0kB bounce:0kB free_pcp:368kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:46:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Jul 18 08:46:19 oak-gw06 kernel: Node 0 Normal free:2640220kB min:55536kB low:69420kB high:83304kB active_anon:106904kB inactive_anon:120384kB active_file:123756kB inactive_file:6348424kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:84632kB writeback:24868kB mapped:36144kB shmem:98692kB slab_reclaimable:169428kB slab_unreclaimable:2911844kB kernel_stack:4704kB pagetables:5248kB unstable:0kB bounce:0kB free_pcp:1212kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 18 08:46:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Jul 18 08:46:19 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Jul 18 08:46:19 oak-gw06 kernel: Node 0 DMA32: 7902*4kB (UEM) 9547*8kB (UEM) 913*16kB (UEM) 2851*32kB (UEM) 1353*64kB (UEM) 1563*128kB (UM) 237*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 561152kB Jul 18 08:46:19 oak-gw06 kernel: Node 0 Normal: 33558*4kB (UEM) 37410*8kB (UEM) 4912*16kB (UEM) 13447*32kB (UEM) 8511*64kB (UEM) 7377*128kB (UEM) 809*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2638472kB Jul 18 08:46:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 18 08:46:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 18 08:46:19 oak-gw06 kernel: 2005349 total pagecache pages Jul 18 08:46:19 oak-gw06 kernel: 0 pages in swap cache Jul 18 08:46:19 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 18 08:46:19 oak-gw06 kernel: Free swap = 4194300kB Jul 18 08:46:19 oak-gw06 kernel: Total swap = 4194300kB Jul 18 08:46:19 oak-gw06 kernel: 4194203 pages RAM Jul 18 08:46:19 oak-gw06 kernel: 0 pages HighMem/MovableOnly Jul 18 08:46:19 oak-gw06 kernel: 127313 pages reserved Jul 24 12:21:45 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3126:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds Jul 24 12:21:45 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3189:kiblnd_check_conns()) Timed out RDMA with 10.0.2.101@o2ib5 (51): c: 0, oc: 0, rc: 8 Jul 24 12:21:45 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1500924053/real 1500924105] req@ffff8801243d9e00 x1566265996650512/t0(0) o400->oak-OST0028-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1500924809 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jul 24 12:21:45 oak-gw06 kernel: Lustre: oak-OST001e-osc-ffff88041b99c000: Connection to oak-OST001e (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jul 24 12:21:45 oak-gw06 kernel: Lustre: Skipped 2 previous similar messages Jul 24 12:21:45 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 88 previous similar messages Jul 24 12:23:24 oak-gw06 kernel: Lustre: 1767:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1500924053/real 1500924204] req@ffff8801af79ad00 x1566265996649936/t0(0) o400->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1500924809 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jul 24 12:23:24 oak-gw06 kernel: Lustre: oak-OST0006-osc-ffff88041b99c000: Connection to oak-OST0006 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jul 24 12:23:24 oak-gw06 kernel: Lustre: Skipped 5 previous similar messages Jul 24 12:23:24 oak-gw06 kernel: Lustre: 1767:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 89 previous similar messages Jul 24 12:28:25 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 12:28:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500924205, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d800/0xf077f1a829b563e4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a204513 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 12:28:25 oak-gw06 kernel: LustreError: 27271:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6240) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 12:28:25 oak-gw06 kernel: LustreError: 27271:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6240) refcount = 2 Jul 24 12:28:25 oak-gw06 kernel: LustreError: 27271:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 12:28:25 oak-gw06 kernel: LustreError: 27271:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d800/0xf077f1a829b563e4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a204513 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 12:28:25 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 12:28:25 oak-gw06 kernel: Lustre: Skipped 13 previous similar messages Jul 24 12:30:51 oak-gw06 kernel: Lustre: oak-OST0014-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jul 24 12:30:54 oak-gw06 kernel: Lustre: oak-OST0028-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jul 24 12:30:54 oak-gw06 kernel: Lustre: Skipped 3 previous similar messages Jul 24 12:31:03 oak-gw06 kernel: Lustre: oak-OST0016-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Jul 24 12:31:03 oak-gw06 kernel: Lustre: Skipped 3 previous similar messages Jul 24 12:33:29 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1500924053/real 1500924053] req@ffff8801af79b300 x1566265996649872/t0(0) o400->oak-OST0000-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1500924809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 24 12:33:29 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 24 12:33:33 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 12:33:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500924513, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f200/0xf077f1a829b56400 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a20c527 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 12:33:33 oak-gw06 kernel: LustreError: 27354:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88041923d300) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 12:33:33 oak-gw06 kernel: LustreError: 27354:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041923d300) refcount = 2 Jul 24 12:33:33 oak-gw06 kernel: LustreError: 27354:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 12:33:33 oak-gw06 kernel: LustreError: 27354:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f200/0xf077f1a829b56400 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a20c527 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 12:33:33 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 12:38:40 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 12:38:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500924820, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56dc00/0xf077f1a829b5641c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a214128 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 12:38:40 oak-gw06 kernel: LustreError: 27357:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 12:38:40 oak-gw06 kernel: LustreError: 27357:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6c00) refcount = 2 Jul 24 12:38:40 oak-gw06 kernel: LustreError: 27357:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 12:38:40 oak-gw06 kernel: LustreError: 27357:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56dc00/0xf077f1a829b5641c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a214128 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 12:38:40 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 12:43:47 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 12:43:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500925127, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d000/0xf077f1a829b56438 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a21b90f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 12:43:47 oak-gw06 kernel: LustreError: 27368:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6240) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 12:43:47 oak-gw06 kernel: LustreError: 27368:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6240) refcount = 2 Jul 24 12:43:47 oak-gw06 kernel: LustreError: 27368:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 12:43:47 oak-gw06 kernel: LustreError: 27368:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d000/0xf077f1a829b56438 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a21b90f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 12:43:47 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 12:48:56 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 12:48:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500925436, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d000/0xf077f1a829b56454 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a22398c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 12:48:56 oak-gw06 kernel: LustreError: 27371:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 12:48:56 oak-gw06 kernel: LustreError: 27371:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6e40) refcount = 2 Jul 24 12:48:56 oak-gw06 kernel: LustreError: 27371:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 12:48:56 oak-gw06 kernel: LustreError: 27371:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d000/0xf077f1a829b56454 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a22398c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 12:48:56 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 12:54:05 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 12:54:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500925745, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f400/0xf077f1a829b56470 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a22bcc5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 12:54:05 oak-gw06 kernel: LustreError: 27381:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6600) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 12:54:05 oak-gw06 kernel: LustreError: 27381:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6600) refcount = 2 Jul 24 12:54:05 oak-gw06 kernel: LustreError: 27381:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 12:54:05 oak-gw06 kernel: LustreError: 27381:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f400/0xf077f1a829b56470 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a22bcc5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 12:54:05 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 12:59:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 12:59:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500926054, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b5648c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a233d96 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 12:59:14 oak-gw06 kernel: LustreError: 27384:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 12:59:14 oak-gw06 kernel: LustreError: 27384:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6b40) refcount = 2 Jul 24 12:59:14 oak-gw06 kernel: LustreError: 27384:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 12:59:14 oak-gw06 kernel: LustreError: 27384:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b5648c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a233d96 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:04:21 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 13:04:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500926361, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b564a8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a23b568 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:04:21 oak-gw06 kernel: LustreError: 27426:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e69c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 13:04:21 oak-gw06 kernel: LustreError: 27426:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e69c0) refcount = 2 Jul 24 13:04:21 oak-gw06 kernel: LustreError: 27426:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 13:04:21 oak-gw06 kernel: LustreError: 27426:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b564a8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a23b568 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:04:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 13:04:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 13:09:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 13:09:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500926671, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d200/0xf077f1a829b564c4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a243797 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:09:31 oak-gw06 kernel: LustreError: 27429:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 13:09:31 oak-gw06 kernel: LustreError: 27429:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6b40) refcount = 2 Jul 24 13:09:31 oak-gw06 kernel: LustreError: 27429:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 13:09:31 oak-gw06 kernel: LustreError: 27429:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d200/0xf077f1a829b564c4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a243797 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:14:39 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 13:14:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500926979, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c200/0xf077f1a829b564e0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a24b4a2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:14:39 oak-gw06 kernel: LustreError: 27439:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 13:14:39 oak-gw06 kernel: LustreError: 27439:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6000) refcount = 2 Jul 24 13:14:39 oak-gw06 kernel: LustreError: 27439:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 13:14:39 oak-gw06 kernel: LustreError: 27439:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c200/0xf077f1a829b564e0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a24b4a2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:14:39 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 13:14:39 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 13:19:46 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 13:19:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500927286, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c800/0xf077f1a829b564fc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a252da1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:19:46 oak-gw06 kernel: LustreError: 27442:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6840) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 13:19:46 oak-gw06 kernel: LustreError: 27442:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6840) refcount = 2 Jul 24 13:19:46 oak-gw06 kernel: LustreError: 27442:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 13:19:46 oak-gw06 kernel: LustreError: 27442:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c800/0xf077f1a829b564fc lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a252da1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:24:52 oak-gw06 kernel: LustreError: 27451:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6f00) refcount = 2 Jul 24 13:24:52 oak-gw06 kernel: LustreError: 27451:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 13:24:52 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 13:24:52 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 13:30:01 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 13:30:01 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 13:30:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500927901, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c000/0xf077f1a829b56534 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a2621a5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:30:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 13:30:01 oak-gw06 kernel: LustreError: 27461:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 13:30:01 oak-gw06 kernel: LustreError: 27461:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 13:30:01 oak-gw06 kernel: LustreError: 27461:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6d80) refcount = 2 Jul 24 13:30:01 oak-gw06 kernel: LustreError: 27461:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 13:30:01 oak-gw06 kernel: LustreError: 27461:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c000/0xf077f1a829b56534 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a2621a5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:30:01 oak-gw06 kernel: LustreError: 27461:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 13:35:08 oak-gw06 kernel: LustreError: 27464:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6cc0) refcount = 2 Jul 24 13:35:08 oak-gw06 kernel: LustreError: 27464:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 13:35:08 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 13:35:08 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 13:40:18 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 13:40:18 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 13:40:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500928518, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b5656c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a271a33 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:40:18 oak-gw06 kernel: LustreError: 27474:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6600) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 13:40:18 oak-gw06 kernel: LustreError: 27474:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 13:40:18 oak-gw06 kernel: LustreError: 27474:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6600) refcount = 2 Jul 24 13:40:18 oak-gw06 kernel: LustreError: 27474:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 13:40:18 oak-gw06 kernel: LustreError: 27474:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b5656c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a271a33 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:40:18 oak-gw06 kernel: LustreError: 27474:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 13:40:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 13:45:27 oak-gw06 kernel: LustreError: 27477:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6c00) refcount = 2 Jul 24 13:45:27 oak-gw06 kernel: LustreError: 27477:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 13:45:27 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 13:45:27 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 13:50:35 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 13:50:35 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 13:50:35 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500929135, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d800/0xf077f1a829b565ab lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a28117f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:50:35 oak-gw06 kernel: LustreError: 27487:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 13:50:35 oak-gw06 kernel: LustreError: 27487:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 13:50:35 oak-gw06 kernel: LustreError: 27487:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6000) refcount = 2 Jul 24 13:50:35 oak-gw06 kernel: LustreError: 27487:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 13:50:35 oak-gw06 kernel: LustreError: 27487:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d800/0xf077f1a829b565ab lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a28117f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 13:50:35 oak-gw06 kernel: LustreError: 27487:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 13:50:35 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 13:55:43 oak-gw06 kernel: LustreError: 27490:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6240) refcount = 2 Jul 24 13:55:43 oak-gw06 kernel: LustreError: 27490:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 13:55:43 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 13:55:43 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 14:00:52 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 14:00:52 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 14:00:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500929752, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f800/0xf077f1a829b565e3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a29096c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 14:00:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 14:00:52 oak-gw06 kernel: LustreError: 27500:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 14:00:52 oak-gw06 kernel: LustreError: 27500:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 14:00:52 oak-gw06 kernel: LustreError: 27500:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6180) refcount = 2 Jul 24 14:00:52 oak-gw06 kernel: LustreError: 27500:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 14:00:52 oak-gw06 kernel: LustreError: 27500:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f800/0xf077f1a829b565e3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a29096c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 14:00:52 oak-gw06 kernel: LustreError: 27500:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 14:06:01 oak-gw06 kernel: LustreError: 27537:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6c00) refcount = 2 Jul 24 14:06:01 oak-gw06 kernel: LustreError: 27537:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 14:06:01 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 14:06:01 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 14:11:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 14:11:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 14:11:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500930371, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c000/0xf077f1a829b5661b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a2a0278 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 14:11:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 14:11:11 oak-gw06 kernel: LustreError: 27548:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 14:11:11 oak-gw06 kernel: LustreError: 27548:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 14:11:11 oak-gw06 kernel: LustreError: 27548:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6900) refcount = 2 Jul 24 14:11:11 oak-gw06 kernel: LustreError: 27548:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 14:11:11 oak-gw06 kernel: LustreError: 27548:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c000/0xf077f1a829b5661b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a2a0278 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 14:11:11 oak-gw06 kernel: LustreError: 27548:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 14:16:16 oak-gw06 kernel: LustreError: 27551:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6cc0) refcount = 2 Jul 24 14:16:16 oak-gw06 kernel: LustreError: 27551:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 14:16:16 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 14:16:16 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 14:21:21 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 14:21:21 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 14:21:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500930981, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c800/0xf077f1a829b56653 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a2aef28 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 14:21:21 oak-gw06 kernel: LustreError: 27561:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 14:21:21 oak-gw06 kernel: LustreError: 27561:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 14:21:21 oak-gw06 kernel: LustreError: 27561:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6f00) refcount = 2 Jul 24 14:21:21 oak-gw06 kernel: LustreError: 27561:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 14:21:21 oak-gw06 kernel: LustreError: 27561:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c800/0xf077f1a829b56653 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a2aef28 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 14:21:21 oak-gw06 kernel: LustreError: 27561:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 14:21:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 14:26:31 oak-gw06 kernel: LustreError: 27565:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6c00) refcount = 2 Jul 24 14:26:31 oak-gw06 kernel: LustreError: 27565:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 14:26:31 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 14:26:31 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 14:31:37 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 14:31:37 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 14:31:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500931597, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56cc00/0xf077f1a829b5668b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a2be46e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 14:31:37 oak-gw06 kernel: LustreError: 27575:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6300) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 14:31:37 oak-gw06 kernel: LustreError: 27575:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 14:31:37 oak-gw06 kernel: LustreError: 27575:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6300) refcount = 2 Jul 24 14:31:37 oak-gw06 kernel: LustreError: 27575:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 14:31:37 oak-gw06 kernel: LustreError: 27575:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56cc00/0xf077f1a829b5668b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a2be46e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 14:31:37 oak-gw06 kernel: LustreError: 27575:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 14:31:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 14:36:46 oak-gw06 kernel: LustreError: 27579:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e69c0) refcount = 2 Jul 24 14:36:46 oak-gw06 kernel: LustreError: 27579:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 14:36:46 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 14:36:46 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 14:41:55 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 14:41:55 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 14:41:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500932215, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b566c3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a2cdc31 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 14:41:55 oak-gw06 kernel: LustreError: 27589:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 14:41:55 oak-gw06 kernel: LustreError: 27589:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 14:41:55 oak-gw06 kernel: LustreError: 27589:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6c00) refcount = 2 Jul 24 14:41:55 oak-gw06 kernel: LustreError: 27589:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 14:41:55 oak-gw06 kernel: LustreError: 27589:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b566c3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a2cdc31 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 14:41:55 oak-gw06 kernel: LustreError: 27589:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 14:41:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 14:47:00 oak-gw06 kernel: LustreError: 27593:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6a80) refcount = 2 Jul 24 14:47:00 oak-gw06 kernel: LustreError: 27593:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 14:47:00 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 14:47:00 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 14:52:10 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 14:52:10 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 14:52:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500932830, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f600/0xf077f1a829b566fb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a2dd097 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 14:52:10 oak-gw06 kernel: LustreError: 27604:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 14:52:10 oak-gw06 kernel: LustreError: 27604:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 14:52:10 oak-gw06 kernel: LustreError: 27604:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6f00) refcount = 2 Jul 24 14:52:10 oak-gw06 kernel: LustreError: 27604:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 14:52:10 oak-gw06 kernel: LustreError: 27604:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f600/0xf077f1a829b566fb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a2dd097 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 14:52:10 oak-gw06 kernel: LustreError: 27604:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 14:52:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 14:57:19 oak-gw06 kernel: LustreError: 27607:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e66c0) refcount = 2 Jul 24 14:57:19 oak-gw06 kernel: LustreError: 27607:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 14:57:19 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 14:57:19 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 15:02:26 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 15:02:26 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 15:02:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500933446, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ea00/0xf077f1a829b56733 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a2ec5dd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 15:02:26 oak-gw06 kernel: LustreError: 27650:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 15:02:26 oak-gw06 kernel: LustreError: 27650:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 15:02:26 oak-gw06 kernel: LustreError: 27650:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6e40) refcount = 2 Jul 24 15:02:26 oak-gw06 kernel: LustreError: 27650:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 15:02:26 oak-gw06 kernel: LustreError: 27650:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ea00/0xf077f1a829b56733 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a2ec5dd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 15:02:26 oak-gw06 kernel: LustreError: 27650:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 15:02:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 15:07:35 oak-gw06 kernel: LustreError: 27653:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10780) refcount = 2 Jul 24 15:07:35 oak-gw06 kernel: LustreError: 27653:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 15:07:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 15:07:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 15:12:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 15:12:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 15:12:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500934064, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f800/0xf077f1a829b56772 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a2fbee2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 15:12:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 15:12:44 oak-gw06 kernel: LustreError: 27663:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10600) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 15:12:44 oak-gw06 kernel: LustreError: 27663:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 15:12:44 oak-gw06 kernel: LustreError: 27663:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10600) refcount = 2 Jul 24 15:12:44 oak-gw06 kernel: LustreError: 27663:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 15:12:44 oak-gw06 kernel: LustreError: 27663:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f800/0xf077f1a829b56772 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a2fbee2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 15:12:44 oak-gw06 kernel: LustreError: 27663:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 15:17:49 oak-gw06 kernel: LustreError: 27666:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10e40) refcount = 2 Jul 24 15:17:49 oak-gw06 kernel: LustreError: 27666:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 15:17:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 15:17:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 15:22:58 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 15:22:58 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 15:22:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500934678, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f400/0xf077f1a829b567aa lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a30b157 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 15:22:58 oak-gw06 kernel: LustreError: 27676:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 15:22:58 oak-gw06 kernel: LustreError: 27676:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 15:22:58 oak-gw06 kernel: LustreError: 27676:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10f00) refcount = 2 Jul 24 15:22:58 oak-gw06 kernel: LustreError: 27676:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 15:22:58 oak-gw06 kernel: LustreError: 27676:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f400/0xf077f1a829b567aa lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a30b157 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 15:22:58 oak-gw06 kernel: LustreError: 27676:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 15:22:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 15:28:07 oak-gw06 kernel: LustreError: 27679:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bb641480) refcount = 2 Jul 24 15:28:07 oak-gw06 kernel: LustreError: 27679:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 15:28:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 15:28:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 15:33:12 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 15:33:12 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 15:33:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500935292, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73000/0xf077f1a829b567e9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a31a38d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 15:33:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 15:33:12 oak-gw06 kernel: LustreError: 27689:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802bb641d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 15:33:12 oak-gw06 kernel: LustreError: 27689:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 15:33:12 oak-gw06 kernel: LustreError: 27689:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bb641d80) refcount = 2 Jul 24 15:33:12 oak-gw06 kernel: LustreError: 27689:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 15:33:12 oak-gw06 kernel: LustreError: 27689:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73000/0xf077f1a829b567e9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a31a38d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 15:33:12 oak-gw06 kernel: LustreError: 27689:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 15:38:21 oak-gw06 kernel: LustreError: 27692:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880293b2c000) refcount = 2 Jul 24 15:38:21 oak-gw06 kernel: LustreError: 27692:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 15:38:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 15:38:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 15:43:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 15:43:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 15:43:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500935908, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73a00/0xf077f1a829b56821 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a329919 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 15:43:28 oak-gw06 kernel: LustreError: 27702:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880293b2ca80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 15:43:28 oak-gw06 kernel: LustreError: 27702:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 15:43:28 oak-gw06 kernel: LustreError: 27702:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880293b2ca80) refcount = 2 Jul 24 15:43:28 oak-gw06 kernel: LustreError: 27702:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 15:43:28 oak-gw06 kernel: LustreError: 27702:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73a00/0xf077f1a829b56821 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a329919 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 15:43:28 oak-gw06 kernel: LustreError: 27702:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 15:43:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 15:48:36 oak-gw06 kernel: LustreError: 27706:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bb641780) refcount = 2 Jul 24 15:48:36 oak-gw06 kernel: LustreError: 27706:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 15:48:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 15:48:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 15:53:45 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 15:53:45 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 15:53:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500936525, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72e00/0xf077f1a829b56859 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a338f77 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 15:53:45 oak-gw06 kernel: LustreError: 27717:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802bb641c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 15:53:45 oak-gw06 kernel: LustreError: 27717:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 15:53:45 oak-gw06 kernel: LustreError: 27717:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bb641c00) refcount = 2 Jul 24 15:53:45 oak-gw06 kernel: LustreError: 27717:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 15:53:45 oak-gw06 kernel: LustreError: 27717:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72e00/0xf077f1a829b56859 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a338f77 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 15:53:45 oak-gw06 kernel: LustreError: 27717:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 15:53:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 15:58:53 oak-gw06 kernel: LustreError: 27720:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880293b2c600) refcount = 2 Jul 24 15:58:53 oak-gw06 kernel: LustreError: 27720:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 15:58:53 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 15:58:53 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 16:04:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 16:04:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 16:04:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500937142, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73e00/0xf077f1a829b56891 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a348637 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 16:04:02 oak-gw06 kernel: LustreError: 27763:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880293b2c480) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 16:04:02 oak-gw06 kernel: LustreError: 27763:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 16:04:02 oak-gw06 kernel: LustreError: 27763:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880293b2c480) refcount = 2 Jul 24 16:04:02 oak-gw06 kernel: LustreError: 27763:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 16:04:02 oak-gw06 kernel: LustreError: 27763:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73e00/0xf077f1a829b56891 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a348637 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 16:04:02 oak-gw06 kernel: LustreError: 27763:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 16:04:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 16:09:08 oak-gw06 kernel: LustreError: 27766:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880293b2c840) refcount = 2 Jul 24 16:09:08 oak-gw06 kernel: LustreError: 27766:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 16:09:08 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 16:09:08 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 16:14:18 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 16:14:18 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 16:14:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500937758, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70600/0xf077f1a829b568c9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a357ba7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 16:14:18 oak-gw06 kernel: LustreError: 27776:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880293b2c9c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 16:14:18 oak-gw06 kernel: LustreError: 27776:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 16:14:18 oak-gw06 kernel: LustreError: 27776:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880293b2c9c0) refcount = 2 Jul 24 16:14:18 oak-gw06 kernel: LustreError: 27776:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 16:14:18 oak-gw06 kernel: LustreError: 27776:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70600/0xf077f1a829b568c9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a357ba7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 16:14:18 oak-gw06 kernel: LustreError: 27776:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 16:14:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 16:19:27 oak-gw06 kernel: LustreError: 27779:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88008e7bd780) refcount = 2 Jul 24 16:19:27 oak-gw06 kernel: LustreError: 27779:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 16:19:27 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 16:19:27 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 16:24:35 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 16:24:35 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 16:24:35 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500938375, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72000/0xf077f1a829b56908 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a3671f0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 16:24:35 oak-gw06 kernel: LustreError: 27790:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88008e7bd540) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 16:24:35 oak-gw06 kernel: LustreError: 27790:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 16:24:35 oak-gw06 kernel: LustreError: 27790:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88008e7bd540) refcount = 2 Jul 24 16:24:35 oak-gw06 kernel: LustreError: 27790:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 16:24:35 oak-gw06 kernel: LustreError: 27790:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72000/0xf077f1a829b56908 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a3671f0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 16:24:35 oak-gw06 kernel: LustreError: 27790:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 16:24:35 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 16:29:44 oak-gw06 kernel: LustreError: 27793:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880293b2c180) refcount = 2 Jul 24 16:29:44 oak-gw06 kernel: LustreError: 27793:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 16:29:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 16:29:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 16:34:52 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 16:34:52 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 16:34:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500938992, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70c00/0xf077f1a829b56940 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a376990 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 16:34:52 oak-gw06 kernel: LustreError: 27804:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880293b2cc00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 16:34:52 oak-gw06 kernel: LustreError: 27804:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 16:34:52 oak-gw06 kernel: LustreError: 27804:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880293b2cc00) refcount = 2 Jul 24 16:34:52 oak-gw06 kernel: LustreError: 27804:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 16:34:52 oak-gw06 kernel: LustreError: 27804:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70c00/0xf077f1a829b56940 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a376990 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 16:34:52 oak-gw06 kernel: LustreError: 27804:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 16:34:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 16:40:01 oak-gw06 kernel: LustreError: 27814:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88008e7bdb40) refcount = 2 Jul 24 16:40:01 oak-gw06 kernel: LustreError: 27814:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 16:40:01 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 16:40:01 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 16:45:09 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 16:45:09 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 16:45:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500939609, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72c00/0xf077f1a829b5697f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a385fa8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 16:45:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 16:45:09 oak-gw06 kernel: LustreError: 27817:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880293b2c540) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 16:45:09 oak-gw06 kernel: LustreError: 27817:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 16:45:09 oak-gw06 kernel: LustreError: 27817:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880293b2c540) refcount = 2 Jul 24 16:45:09 oak-gw06 kernel: LustreError: 27817:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 16:45:09 oak-gw06 kernel: LustreError: 27817:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72c00/0xf077f1a829b5697f lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a385fa8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 16:45:09 oak-gw06 kernel: LustreError: 27817:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 16:50:16 oak-gw06 kernel: LustreError: 27827:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880293b2c300) refcount = 2 Jul 24 16:50:16 oak-gw06 kernel: LustreError: 27827:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 16:50:16 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 16:50:16 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 16:55:22 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 16:55:22 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 16:55:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500940222, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73a00/0xf077f1a829b569be lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a395358 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 16:55:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 16:55:22 oak-gw06 kernel: LustreError: 27830:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880293b2c240) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 16:55:22 oak-gw06 kernel: LustreError: 27830:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 16:55:22 oak-gw06 kernel: LustreError: 27830:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880293b2c240) refcount = 2 Jul 24 16:55:22 oak-gw06 kernel: LustreError: 27830:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 16:55:22 oak-gw06 kernel: LustreError: 27830:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73a00/0xf077f1a829b569be lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a395358 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 16:55:22 oak-gw06 kernel: LustreError: 27830:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 17:00:28 oak-gw06 kernel: LustreError: 27840:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880293b2c840) refcount = 2 Jul 24 17:00:28 oak-gw06 kernel: LustreError: 27840:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 17:00:28 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 17:00:28 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 17:05:33 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 17:05:33 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 17:05:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500940833, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70e00/0xf077f1a829b569f6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a3a4492 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 17:05:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 17:05:33 oak-gw06 kernel: LustreError: 27875:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88008e7bd840) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 17:05:34 oak-gw06 kernel: LustreError: 27875:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 17:05:34 oak-gw06 kernel: LustreError: 27875:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88008e7bd840) refcount = 2 Jul 24 17:05:34 oak-gw06 kernel: LustreError: 27875:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 17:05:34 oak-gw06 kernel: LustreError: 27875:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70e00/0xf077f1a829b569f6 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a3a4492 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 17:05:34 oak-gw06 kernel: LustreError: 27875:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 17:10:41 oak-gw06 kernel: LustreError: 27885:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880293b2cb40) refcount = 2 Jul 24 17:10:41 oak-gw06 kernel: LustreError: 27885:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 17:10:41 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 17:10:41 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 17:15:50 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 17:15:50 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 17:15:50 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500941450, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71200/0xf077f1a829b56a3c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a3b3ae2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 17:15:50 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 17:15:50 oak-gw06 kernel: LustreError: 27888:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880009e6ca80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 17:15:50 oak-gw06 kernel: LustreError: 27888:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 17:15:50 oak-gw06 kernel: LustreError: 27888:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880009e6ca80) refcount = 2 Jul 24 17:15:50 oak-gw06 kernel: LustreError: 27888:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 17:15:50 oak-gw06 kernel: LustreError: 27888:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71200/0xf077f1a829b56a3c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a3b3ae2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 17:15:50 oak-gw06 kernel: LustreError: 27888:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 17:20:56 oak-gw06 kernel: LustreError: 27899:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880009e6c000) refcount = 2 Jul 24 17:20:56 oak-gw06 kernel: LustreError: 27899:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 17:20:56 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 17:20:56 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 17:26:04 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 17:26:04 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 17:26:04 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500942064, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72800/0xf077f1a829b56a74 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a3c2f33 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 17:26:04 oak-gw06 kernel: LustreError: 27903:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800541840c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 17:26:04 oak-gw06 kernel: LustreError: 27903:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 17:26:04 oak-gw06 kernel: LustreError: 27903:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800541840c0) refcount = 2 Jul 24 17:26:04 oak-gw06 kernel: LustreError: 27903:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 17:26:04 oak-gw06 kernel: LustreError: 27903:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72800/0xf077f1a829b56a74 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a3c2f33 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 17:26:04 oak-gw06 kernel: LustreError: 27903:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 17:26:04 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 17:31:12 oak-gw06 kernel: LustreError: 27914:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880054184300) refcount = 2 Jul 24 17:31:12 oak-gw06 kernel: LustreError: 27914:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 17:31:12 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 17:31:12 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 17:36:21 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 17:36:21 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 17:36:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500942681, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71c00/0xf077f1a829b56aac lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a3d25bb expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 17:36:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 17:36:21 oak-gw06 kernel: LustreError: 27917:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880009e6ce40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 17:36:21 oak-gw06 kernel: LustreError: 27917:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 17:36:21 oak-gw06 kernel: LustreError: 27917:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880009e6ce40) refcount = 2 Jul 24 17:36:21 oak-gw06 kernel: LustreError: 27917:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 17:36:21 oak-gw06 kernel: LustreError: 27917:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71c00/0xf077f1a829b56aac lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a3d25bb expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 17:36:21 oak-gw06 kernel: LustreError: 27917:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 17:41:26 oak-gw06 kernel: LustreError: 27927:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880009e6c780) refcount = 2 Jul 24 17:41:26 oak-gw06 kernel: LustreError: 27927:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 17:41:26 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 17:41:26 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 17:46:33 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 17:46:33 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 17:46:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500943293, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70800/0xf077f1a829b56aeb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a3e1344 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 17:46:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 17:46:33 oak-gw06 kernel: LustreError: 27930:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880054184300) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 17:46:33 oak-gw06 kernel: LustreError: 27930:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 17:46:33 oak-gw06 kernel: LustreError: 27930:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880054184300) refcount = 2 Jul 24 17:46:33 oak-gw06 kernel: LustreError: 27930:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 17:46:33 oak-gw06 kernel: LustreError: 27930:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70800/0xf077f1a829b56aeb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a3e1344 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 17:46:33 oak-gw06 kernel: LustreError: 27930:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 17:51:39 oak-gw06 kernel: LustreError: 27940:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880054184e40) refcount = 2 Jul 24 17:51:39 oak-gw06 kernel: LustreError: 27940:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 17:51:39 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 17:51:39 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 17:56:45 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 17:56:45 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 17:56:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500943905, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72c00/0xf077f1a829b56b2a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a3f02d3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 17:56:45 oak-gw06 kernel: LustreError: 27943:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880009e6c780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 17:56:45 oak-gw06 kernel: LustreError: 27943:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 17:56:45 oak-gw06 kernel: LustreError: 27943:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880009e6c780) refcount = 2 Jul 24 17:56:45 oak-gw06 kernel: LustreError: 27943:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 17:56:45 oak-gw06 kernel: LustreError: 27943:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72c00/0xf077f1a829b56b2a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a3f02d3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 17:56:45 oak-gw06 kernel: LustreError: 27943:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 17:56:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 18:01:55 oak-gw06 kernel: LustreError: 27985:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880009e6c6c0) refcount = 2 Jul 24 18:01:55 oak-gw06 kernel: LustreError: 27985:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 18:01:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 18:01:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 18:07:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 18:07:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 18:07:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500944522, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73e00/0xf077f1a829b56b69 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a3ff9cb expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 18:07:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 18:07:02 oak-gw06 kernel: LustreError: 27988:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880054184a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 18:07:02 oak-gw06 kernel: LustreError: 27988:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 18:07:02 oak-gw06 kernel: LustreError: 27988:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880054184a80) refcount = 2 Jul 24 18:07:02 oak-gw06 kernel: LustreError: 27988:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 18:07:02 oak-gw06 kernel: LustreError: 27988:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73e00/0xf077f1a829b56b69 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a3ff9cb expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 18:07:02 oak-gw06 kernel: LustreError: 27988:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 18:12:07 oak-gw06 kernel: LustreError: 27998:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880054184840) refcount = 2 Jul 24 18:12:07 oak-gw06 kernel: LustreError: 27998:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 18:12:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 18:12:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 18:17:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 18:17:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 18:17:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500945136, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d600/0xf077f1a829b56ba8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a40ec5c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 18:17:16 oak-gw06 kernel: LustreError: 28001:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 18:17:16 oak-gw06 kernel: LustreError: 28001:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 18:17:16 oak-gw06 kernel: LustreError: 28001:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10cc0) refcount = 2 Jul 24 18:17:16 oak-gw06 kernel: LustreError: 28001:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 18:17:16 oak-gw06 kernel: LustreError: 28001:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d600/0xf077f1a829b56ba8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a40ec5c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 18:17:16 oak-gw06 kernel: LustreError: 28001:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 18:17:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 18:22:24 oak-gw06 kernel: LustreError: 28011:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10300) refcount = 2 Jul 24 18:22:24 oak-gw06 kernel: LustreError: 28011:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 18:22:24 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 18:22:24 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 18:27:33 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 18:27:33 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 18:27:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500945753, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56da00/0xf077f1a829b56be7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a41e4f8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 18:27:33 oak-gw06 kernel: LustreError: 28014:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 18:27:33 oak-gw06 kernel: LustreError: 28014:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 18:27:33 oak-gw06 kernel: LustreError: 28014:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10f00) refcount = 2 Jul 24 18:27:33 oak-gw06 kernel: LustreError: 28014:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 18:27:33 oak-gw06 kernel: LustreError: 28014:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56da00/0xf077f1a829b56be7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a41e4f8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 18:27:33 oak-gw06 kernel: LustreError: 28014:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 18:27:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 18:32:41 oak-gw06 kernel: LustreError: 28024:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10540) refcount = 2 Jul 24 18:32:41 oak-gw06 kernel: LustreError: 28024:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 18:32:41 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 18:32:41 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 18:37:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 18:37:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 18:37:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500946369, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d600/0xf077f1a829b56c1f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a42d957 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 18:37:49 oak-gw06 kernel: LustreError: 28027:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 18:37:49 oak-gw06 kernel: LustreError: 28027:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 18:37:49 oak-gw06 kernel: LustreError: 28027:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10b40) refcount = 2 Jul 24 18:37:49 oak-gw06 kernel: LustreError: 28027:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 18:37:49 oak-gw06 kernel: LustreError: 28027:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d600/0xf077f1a829b56c1f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a42d957 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 18:37:49 oak-gw06 kernel: LustreError: 28027:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 18:37:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 18:42:58 oak-gw06 kernel: LustreError: 28037:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10780) refcount = 2 Jul 24 18:42:58 oak-gw06 kernel: LustreError: 28037:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 18:42:58 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 18:42:58 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 18:48:07 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 18:48:07 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 18:48:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500946987, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b56c57 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a43d343 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 18:48:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 18:48:07 oak-gw06 kernel: LustreError: 28040:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 18:48:07 oak-gw06 kernel: LustreError: 28040:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 18:48:07 oak-gw06 kernel: LustreError: 28040:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10780) refcount = 2 Jul 24 18:48:07 oak-gw06 kernel: LustreError: 28040:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 18:48:07 oak-gw06 kernel: LustreError: 28040:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b56c57 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a43d343 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 18:48:07 oak-gw06 kernel: LustreError: 28040:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 18:53:14 oak-gw06 kernel: LustreError: 28052:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b109c0) refcount = 2 Jul 24 18:53:14 oak-gw06 kernel: LustreError: 28052:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 18:53:14 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 18:53:14 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 18:58:21 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 18:58:21 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 18:58:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500947601, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b56c8f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a44c5b8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 18:58:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 18:58:21 oak-gw06 kernel: LustreError: 28055:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 18:58:21 oak-gw06 kernel: LustreError: 28055:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 18:58:21 oak-gw06 kernel: LustreError: 28055:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10d80) refcount = 2 Jul 24 18:58:21 oak-gw06 kernel: LustreError: 28055:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 18:58:21 oak-gw06 kernel: LustreError: 28055:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b56c8f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a44c5b8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 18:58:21 oak-gw06 kernel: LustreError: 28055:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 19:03:26 oak-gw06 kernel: LustreError: 28097:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10cc0) refcount = 2 Jul 24 19:03:26 oak-gw06 kernel: LustreError: 28097:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 19:03:26 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 19:03:26 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 19:08:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 19:08:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 19:08:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500948211, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ea00/0xf077f1a829b56cc7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a45af12 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 19:08:31 oak-gw06 kernel: LustreError: 28101:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 19:08:31 oak-gw06 kernel: LustreError: 28101:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 19:08:31 oak-gw06 kernel: LustreError: 28101:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10b40) refcount = 2 Jul 24 19:08:31 oak-gw06 kernel: LustreError: 28101:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 19:08:31 oak-gw06 kernel: LustreError: 28101:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ea00/0xf077f1a829b56cc7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a45af12 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 19:08:31 oak-gw06 kernel: LustreError: 28101:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 19:08:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 19:13:40 oak-gw06 kernel: LustreError: 28111:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10540) refcount = 2 Jul 24 19:13:40 oak-gw06 kernel: LustreError: 28111:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 19:13:40 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 19:13:40 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 19:18:50 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 19:18:50 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 19:18:50 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500948830, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d000/0xf077f1a829b56cff lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a46aab0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 19:18:50 oak-gw06 kernel: LustreError: 28114:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 19:18:50 oak-gw06 kernel: LustreError: 28114:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 19:18:50 oak-gw06 kernel: LustreError: 28114:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10900) refcount = 2 Jul 24 19:18:50 oak-gw06 kernel: LustreError: 28114:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 19:18:50 oak-gw06 kernel: LustreError: 28114:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d000/0xf077f1a829b56cff lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a46aab0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 19:18:50 oak-gw06 kernel: LustreError: 28114:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 19:18:50 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 19:23:56 oak-gw06 kernel: LustreError: 28124:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10840) refcount = 2 Jul 24 19:23:56 oak-gw06 kernel: LustreError: 28124:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 19:23:56 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 19:23:56 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 19:29:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 19:29:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 19:29:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500949442, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f000/0xf077f1a829b56d3e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a4799b3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 19:29:02 oak-gw06 kernel: LustreError: 28127:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 19:29:02 oak-gw06 kernel: LustreError: 28127:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 19:29:02 oak-gw06 kernel: LustreError: 28127:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10e40) refcount = 2 Jul 24 19:29:02 oak-gw06 kernel: LustreError: 28127:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 19:29:02 oak-gw06 kernel: LustreError: 28127:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f000/0xf077f1a829b56d3e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a4799b3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 19:29:02 oak-gw06 kernel: LustreError: 28127:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 19:29:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 19:34:09 oak-gw06 kernel: LustreError: 28137:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10240) refcount = 2 Jul 24 19:34:09 oak-gw06 kernel: LustreError: 28137:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 19:34:09 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 19:34:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 19:39:19 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 19:39:19 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 19:39:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500950059, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c800/0xf077f1a829b56d76 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a48902d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 19:39:19 oak-gw06 kernel: LustreError: 28140:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 19:39:19 oak-gw06 kernel: LustreError: 28140:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 19:39:19 oak-gw06 kernel: LustreError: 28140:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10000) refcount = 2 Jul 24 19:39:19 oak-gw06 kernel: LustreError: 28140:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 19:39:19 oak-gw06 kernel: LustreError: 28140:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c800/0xf077f1a829b56d76 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a48902d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 19:39:19 oak-gw06 kernel: LustreError: 28140:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 19:39:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 19:44:25 oak-gw06 kernel: LustreError: 28150:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10e40) refcount = 2 Jul 24 19:44:25 oak-gw06 kernel: LustreError: 28150:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 19:44:25 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 19:44:25 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 19:49:30 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 19:49:30 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 19:49:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500950670, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56da00/0xf077f1a829b56dae lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a497a67 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 19:49:30 oak-gw06 kernel: LustreError: 28153:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 19:49:30 oak-gw06 kernel: LustreError: 28153:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 19:49:30 oak-gw06 kernel: LustreError: 28153:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10f00) refcount = 2 Jul 24 19:49:30 oak-gw06 kernel: LustreError: 28153:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 19:49:30 oak-gw06 kernel: LustreError: 28153:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56da00/0xf077f1a829b56dae lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a497a67 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 19:49:30 oak-gw06 kernel: LustreError: 28153:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 19:49:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 19:54:36 oak-gw06 kernel: LustreError: 28164:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10a80) refcount = 2 Jul 24 19:54:36 oak-gw06 kernel: LustreError: 28164:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 19:54:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 19:54:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 19:59:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 19:59:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 19:59:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500951282, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e000/0xf077f1a829b56de6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a4a678e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 19:59:42 oak-gw06 kernel: LustreError: 28167:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 19:59:42 oak-gw06 kernel: LustreError: 28167:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 19:59:42 oak-gw06 kernel: LustreError: 28167:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10f00) refcount = 2 Jul 24 19:59:42 oak-gw06 kernel: LustreError: 28167:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 19:59:42 oak-gw06 kernel: LustreError: 28167:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e000/0xf077f1a829b56de6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a4a678e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 19:59:42 oak-gw06 kernel: LustreError: 28167:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 19:59:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 20:04:52 oak-gw06 kernel: LustreError: 28209:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10540) refcount = 2 Jul 24 20:04:52 oak-gw06 kernel: LustreError: 28209:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 20:04:52 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 20:04:52 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 20:09:58 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 20:09:58 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 20:09:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500951898, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ce00/0xf077f1a829b56e1e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a4b5d83 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 20:09:58 oak-gw06 kernel: LustreError: 28212:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10240) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 20:09:58 oak-gw06 kernel: LustreError: 28212:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 20:09:58 oak-gw06 kernel: LustreError: 28212:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10240) refcount = 2 Jul 24 20:09:58 oak-gw06 kernel: LustreError: 28212:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 20:09:58 oak-gw06 kernel: LustreError: 28212:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ce00/0xf077f1a829b56e1e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a4b5d83 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 20:09:58 oak-gw06 kernel: LustreError: 28212:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 20:09:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 20:15:07 oak-gw06 kernel: LustreError: 28223:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b103c0) refcount = 2 Jul 24 20:15:07 oak-gw06 kernel: LustreError: 28223:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 20:15:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 20:15:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 20:20:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 20:20:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 20:20:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500952516, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f200/0xf077f1a829b56e56 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a4c5b89 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 20:20:16 oak-gw06 kernel: LustreError: 28234:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 20:20:16 oak-gw06 kernel: LustreError: 28234:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 20:20:16 oak-gw06 kernel: LustreError: 28234:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10780) refcount = 2 Jul 24 20:20:16 oak-gw06 kernel: LustreError: 28234:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 20:20:16 oak-gw06 kernel: LustreError: 28234:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f200/0xf077f1a829b56e56 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a4c5b89 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 20:20:16 oak-gw06 kernel: LustreError: 28234:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 20:20:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 20:25:25 oak-gw06 kernel: LustreError: 28237:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10300) refcount = 2 Jul 24 20:25:25 oak-gw06 kernel: LustreError: 28237:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 20:25:25 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 20:25:25 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 20:30:34 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 20:30:34 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 20:30:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500953134, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ec00/0xf077f1a829b56e8e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a4d5399 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 20:30:34 oak-gw06 kernel: LustreError: 28247:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 20:30:34 oak-gw06 kernel: LustreError: 28247:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 20:30:34 oak-gw06 kernel: LustreError: 28247:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10a80) refcount = 2 Jul 24 20:30:34 oak-gw06 kernel: LustreError: 28247:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 20:30:34 oak-gw06 kernel: LustreError: 28247:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ec00/0xf077f1a829b56e8e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a4d5399 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 20:30:34 oak-gw06 kernel: LustreError: 28247:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 20:30:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 20:35:44 oak-gw06 kernel: LustreError: 28250:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10300) refcount = 2 Jul 24 20:35:44 oak-gw06 kernel: LustreError: 28250:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 20:35:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 20:35:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 20:40:53 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 20:40:53 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 20:40:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500953753, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c000/0xf077f1a829b56ec6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a4e518a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 20:40:53 oak-gw06 kernel: LustreError: 28260:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 20:40:53 oak-gw06 kernel: LustreError: 28260:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 20:40:53 oak-gw06 kernel: LustreError: 28260:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10c00) refcount = 2 Jul 24 20:40:53 oak-gw06 kernel: LustreError: 28260:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 20:40:53 oak-gw06 kernel: LustreError: 28260:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c000/0xf077f1a829b56ec6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a4e518a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 20:40:53 oak-gw06 kernel: LustreError: 28260:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 20:40:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 20:45:58 oak-gw06 kernel: LustreError: 28264:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b103c0) refcount = 2 Jul 24 20:45:58 oak-gw06 kernel: LustreError: 28264:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 20:45:58 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 20:45:58 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 20:51:06 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 20:51:06 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 20:51:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500954366, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e200/0xf077f1a829b56efe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a4f420e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 20:51:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 20:51:06 oak-gw06 kernel: LustreError: 28275:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 20:51:06 oak-gw06 kernel: LustreError: 28275:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 20:51:06 oak-gw06 kernel: LustreError: 28275:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10b40) refcount = 2 Jul 24 20:51:06 oak-gw06 kernel: LustreError: 28275:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 20:51:06 oak-gw06 kernel: LustreError: 28275:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e200/0xf077f1a829b56efe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a4f420e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 20:51:06 oak-gw06 kernel: LustreError: 28275:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 20:56:14 oak-gw06 kernel: LustreError: 28278:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10cc0) refcount = 2 Jul 24 20:56:14 oak-gw06 kernel: LustreError: 28278:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 20:56:14 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 20:56:14 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 21:01:23 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 21:01:23 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 21:01:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500954983, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e800/0xf077f1a829b56f36 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a503a10 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 21:01:23 oak-gw06 kernel: LustreError: 28320:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10540) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 21:01:23 oak-gw06 kernel: LustreError: 28320:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 21:01:23 oak-gw06 kernel: LustreError: 28320:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10540) refcount = 2 Jul 24 21:01:23 oak-gw06 kernel: LustreError: 28320:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 21:01:23 oak-gw06 kernel: LustreError: 28320:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e800/0xf077f1a829b56f36 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a503a10 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 21:01:23 oak-gw06 kernel: LustreError: 28320:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 21:01:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 21:06:31 oak-gw06 kernel: LustreError: 28323:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10e40) refcount = 2 Jul 24 21:06:31 oak-gw06 kernel: LustreError: 28323:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 21:06:31 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 21:06:31 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 21:11:38 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 21:11:38 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 21:11:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500955598, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fc00/0xf077f1a829b56f6e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a512e76 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 21:11:38 oak-gw06 kernel: LustreError: 28333:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10600) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 21:11:38 oak-gw06 kernel: LustreError: 28333:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 21:11:38 oak-gw06 kernel: LustreError: 28333:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10600) refcount = 2 Jul 24 21:11:38 oak-gw06 kernel: LustreError: 28333:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 21:11:38 oak-gw06 kernel: LustreError: 28333:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fc00/0xf077f1a829b56f6e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a512e76 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 21:11:38 oak-gw06 kernel: LustreError: 28333:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 21:11:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 21:16:47 oak-gw06 kernel: LustreError: 28336:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b103c0) refcount = 2 Jul 24 21:16:47 oak-gw06 kernel: LustreError: 28336:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 21:16:47 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 21:16:47 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 21:21:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 21:21:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 21:21:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500956217, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f800/0xf077f1a829b56fa6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a522a99 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 21:21:57 oak-gw06 kernel: LustreError: 28346:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 21:21:57 oak-gw06 kernel: LustreError: 28346:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 21:21:57 oak-gw06 kernel: LustreError: 28346:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10cc0) refcount = 2 Jul 24 21:21:57 oak-gw06 kernel: LustreError: 28346:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 21:21:57 oak-gw06 kernel: LustreError: 28346:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f800/0xf077f1a829b56fa6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a522a99 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 21:21:57 oak-gw06 kernel: LustreError: 28346:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 21:21:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 21:27:03 oak-gw06 kernel: LustreError: 28349:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b109c0) refcount = 2 Jul 24 21:27:03 oak-gw06 kernel: LustreError: 28349:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 21:27:03 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 21:27:03 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 21:32:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 21:32:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 21:32:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500956831, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fe00/0xf077f1a829b56fde lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a531ee3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 21:32:11 oak-gw06 kernel: LustreError: 28359:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 21:32:11 oak-gw06 kernel: LustreError: 28359:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 21:32:11 oak-gw06 kernel: LustreError: 28359:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10900) refcount = 2 Jul 24 21:32:11 oak-gw06 kernel: LustreError: 28359:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 21:32:11 oak-gw06 kernel: LustreError: 28359:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fe00/0xf077f1a829b56fde lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a531ee3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 21:32:11 oak-gw06 kernel: LustreError: 28359:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 21:32:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 21:37:18 oak-gw06 kernel: LustreError: 28362:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10cc0) refcount = 2 Jul 24 21:37:18 oak-gw06 kernel: LustreError: 28362:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 21:37:18 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 21:37:18 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 21:42:27 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 21:42:27 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 21:42:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500957447, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b57016 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a541437 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 21:42:27 oak-gw06 kernel: LustreError: 28372:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 21:42:27 oak-gw06 kernel: LustreError: 28372:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 21:42:27 oak-gw06 kernel: LustreError: 28372:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10000) refcount = 2 Jul 24 21:42:27 oak-gw06 kernel: LustreError: 28372:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 21:42:27 oak-gw06 kernel: LustreError: 28372:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b57016 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a541437 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 21:42:27 oak-gw06 kernel: LustreError: 28372:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 21:42:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 21:47:34 oak-gw06 kernel: LustreError: 28375:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10780) refcount = 2 Jul 24 21:47:34 oak-gw06 kernel: LustreError: 28375:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 21:47:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 21:47:34 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 21:52:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 21:52:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 21:52:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500958061, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c000/0xf077f1a829b5704e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a550643 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 21:52:41 oak-gw06 kernel: LustreError: 28385:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 21:52:41 oak-gw06 kernel: LustreError: 28385:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 21:52:41 oak-gw06 kernel: LustreError: 28385:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10b40) refcount = 2 Jul 24 21:52:41 oak-gw06 kernel: LustreError: 28385:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 21:52:41 oak-gw06 kernel: LustreError: 28385:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c000/0xf077f1a829b5704e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a550643 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 21:52:41 oak-gw06 kernel: LustreError: 28385:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 21:52:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 21:57:50 oak-gw06 kernel: LustreError: 28388:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10d80) refcount = 2 Jul 24 21:57:50 oak-gw06 kernel: LustreError: 28388:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 21:57:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 21:57:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 22:02:55 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 22:02:55 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 22:02:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500958675, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d600/0xf077f1a829b5708d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a55fa4e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 22:02:55 oak-gw06 kernel: LustreError: 28430:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b109c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 22:02:55 oak-gw06 kernel: LustreError: 28430:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 22:02:55 oak-gw06 kernel: LustreError: 28430:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b109c0) refcount = 2 Jul 24 22:02:55 oak-gw06 kernel: LustreError: 28430:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 22:02:55 oak-gw06 kernel: LustreError: 28430:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d600/0xf077f1a829b5708d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a55fa4e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 22:02:55 oak-gw06 kernel: LustreError: 28430:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 22:02:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 22:08:04 oak-gw06 kernel: LustreError: 28433:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10480) refcount = 2 Jul 24 22:08:04 oak-gw06 kernel: LustreError: 28433:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 22:08:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 22:08:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 22:13:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 22:13:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 22:13:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500959294, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326f400/0xf077f1a829b570cc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a56f704 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 22:13:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 22:13:14 oak-gw06 kernel: LustreError: 28443:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 22:13:14 oak-gw06 kernel: LustreError: 28443:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 22:13:14 oak-gw06 kernel: LustreError: 28443:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2f00) refcount = 2 Jul 24 22:13:14 oak-gw06 kernel: LustreError: 28443:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 22:13:14 oak-gw06 kernel: LustreError: 28443:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326f400/0xf077f1a829b570cc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a56f704 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 22:13:14 oak-gw06 kernel: LustreError: 28443:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 22:18:23 oak-gw06 kernel: LustreError: 28446:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2d80) refcount = 2 Jul 24 22:18:23 oak-gw06 kernel: LustreError: 28446:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 22:18:23 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 22:18:23 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 22:23:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 22:23:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 22:23:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500959911, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e200/0xf077f1a829b57104 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a57ef4c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 22:23:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 22:23:31 oak-gw06 kernel: LustreError: 28456:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2840) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 22:23:31 oak-gw06 kernel: LustreError: 28456:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 22:23:31 oak-gw06 kernel: LustreError: 28456:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2840) refcount = 2 Jul 24 22:23:31 oak-gw06 kernel: LustreError: 28456:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 22:23:31 oak-gw06 kernel: LustreError: 28456:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e200/0xf077f1a829b57104 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a57ef4c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 22:23:31 oak-gw06 kernel: LustreError: 28456:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 22:28:36 oak-gw06 kernel: LustreError: 28460:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2840) refcount = 2 Jul 24 22:28:36 oak-gw06 kernel: LustreError: 28460:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 22:28:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 22:28:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 22:33:45 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 22:33:45 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 22:33:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500960525, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326d800/0xf077f1a829b57143 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a58e135 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 22:33:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 22:33:45 oak-gw06 kernel: LustreError: 28471:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2840) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 22:33:45 oak-gw06 kernel: LustreError: 28471:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 22:33:45 oak-gw06 kernel: LustreError: 28471:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2840) refcount = 2 Jul 24 22:33:45 oak-gw06 kernel: LustreError: 28471:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 22:33:45 oak-gw06 kernel: LustreError: 28471:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326d800/0xf077f1a829b57143 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a58e135 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 22:33:45 oak-gw06 kernel: LustreError: 28471:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 22:38:51 oak-gw06 kernel: LustreError: 28474:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2f00) refcount = 2 Jul 24 22:38:51 oak-gw06 kernel: LustreError: 28474:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 22:38:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 22:38:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 22:44:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 22:44:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 22:44:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500961140, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ca00/0xf077f1a829b57182 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a59d4c2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 22:44:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 22:44:00 oak-gw06 kernel: LustreError: 28484:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 22:44:00 oak-gw06 kernel: LustreError: 28484:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 22:44:00 oak-gw06 kernel: LustreError: 28484:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2e40) refcount = 2 Jul 24 22:44:00 oak-gw06 kernel: LustreError: 28484:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 22:44:00 oak-gw06 kernel: LustreError: 28484:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ca00/0xf077f1a829b57182 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a59d4c2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 22:44:00 oak-gw06 kernel: LustreError: 28484:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 22:49:10 oak-gw06 kernel: LustreError: 28487:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2a80) refcount = 2 Jul 24 22:49:10 oak-gw06 kernel: LustreError: 28487:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 22:49:10 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 22:49:10 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 22:54:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 22:54:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 22:54:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500961756, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326fa00/0xf077f1a829b571c1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a5aca01 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 22:54:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 22:54:16 oak-gw06 kernel: LustreError: 28498:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 22:54:16 oak-gw06 kernel: LustreError: 28498:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 22:54:16 oak-gw06 kernel: LustreError: 28498:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2e40) refcount = 2 Jul 24 22:54:16 oak-gw06 kernel: LustreError: 28498:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 22:54:16 oak-gw06 kernel: LustreError: 28498:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326fa00/0xf077f1a829b571c1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a5aca01 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 22:54:16 oak-gw06 kernel: LustreError: 28498:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 22:59:21 oak-gw06 kernel: LustreError: 28501:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2480) refcount = 2 Jul 24 22:59:21 oak-gw06 kernel: LustreError: 28501:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 22:59:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 22:59:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 23:04:29 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 23:04:29 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 23:04:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500962369, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326fa00/0xf077f1a829b571f9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a5bba9a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 23:04:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 23:04:29 oak-gw06 kernel: LustreError: 28543:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d26c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 23:04:29 oak-gw06 kernel: LustreError: 28543:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 23:04:29 oak-gw06 kernel: LustreError: 28543:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d26c0) refcount = 2 Jul 24 23:04:29 oak-gw06 kernel: LustreError: 28543:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 23:04:29 oak-gw06 kernel: LustreError: 28543:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326fa00/0xf077f1a829b571f9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a5bba9a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 23:04:29 oak-gw06 kernel: LustreError: 28543:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 23:09:36 oak-gw06 kernel: LustreError: 28547:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2600) refcount = 2 Jul 24 23:09:36 oak-gw06 kernel: LustreError: 28547:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 23:09:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 23:09:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 23:14:43 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 23:14:43 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 23:14:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500962983, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326f800/0xf077f1a829b57231 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a5caaa0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 23:14:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 23:14:43 oak-gw06 kernel: LustreError: 28559:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 23:14:43 oak-gw06 kernel: LustreError: 28559:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 23:14:43 oak-gw06 kernel: LustreError: 28559:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2900) refcount = 2 Jul 24 23:14:43 oak-gw06 kernel: LustreError: 28559:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 23:14:43 oak-gw06 kernel: LustreError: 28559:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326f800/0xf077f1a829b57231 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a5caaa0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 23:14:43 oak-gw06 kernel: LustreError: 28559:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 23:19:51 oak-gw06 kernel: LustreError: 28562:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2300) refcount = 2 Jul 24 23:19:51 oak-gw06 kernel: LustreError: 28562:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 23:19:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 23:19:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 23:24:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 23:24:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 23:24:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500963597, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326c000/0xf077f1a829b57269 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a5d9dd9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 23:24:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 23:24:57 oak-gw06 kernel: LustreError: 28572:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d29c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 23:24:57 oak-gw06 kernel: LustreError: 28572:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 23:24:57 oak-gw06 kernel: LustreError: 28572:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d29c0) refcount = 2 Jul 24 23:24:57 oak-gw06 kernel: LustreError: 28572:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 23:24:57 oak-gw06 kernel: LustreError: 28572:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326c000/0xf077f1a829b57269 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a5d9dd9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 23:24:57 oak-gw06 kernel: LustreError: 28572:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 23:30:03 oak-gw06 kernel: LustreError: 28582:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2600) refcount = 2 Jul 24 23:30:03 oak-gw06 kernel: LustreError: 28582:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 23:30:03 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 23:30:03 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 23:35:10 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 23:35:10 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 23:35:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500964210, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6c600/0xf077f1a829b572a1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a5e8c26 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 23:35:10 oak-gw06 kernel: LustreError: 28585:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88004f33a540) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 23:35:10 oak-gw06 kernel: LustreError: 28585:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 23:35:10 oak-gw06 kernel: LustreError: 28585:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88004f33a540) refcount = 2 Jul 24 23:35:10 oak-gw06 kernel: LustreError: 28585:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 23:35:10 oak-gw06 kernel: LustreError: 28585:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6c600/0xf077f1a829b572a1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a5e8c26 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 23:35:10 oak-gw06 kernel: LustreError: 28585:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 23:35:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 23:40:19 oak-gw06 kernel: LustreError: 28595:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88004f33a900) refcount = 2 Jul 24 23:40:19 oak-gw06 kernel: LustreError: 28595:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 23:40:19 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 23:40:19 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 23:45:26 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 23:45:26 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 23:45:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500964826, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6da00/0xf077f1a829b572e0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a5f8333 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 23:45:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 23:45:26 oak-gw06 kernel: LustreError: 28598:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88004f33a6c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 23:45:26 oak-gw06 kernel: LustreError: 28598:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 23:45:26 oak-gw06 kernel: LustreError: 28598:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88004f33a6c0) refcount = 2 Jul 24 23:45:26 oak-gw06 kernel: LustreError: 28598:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 23:45:26 oak-gw06 kernel: LustreError: 28598:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6da00/0xf077f1a829b572e0 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a5f8333 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 23:45:26 oak-gw06 kernel: LustreError: 28598:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 24 23:50:34 oak-gw06 kernel: LustreError: 28608:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88004f33a900) refcount = 2 Jul 24 23:50:34 oak-gw06 kernel: LustreError: 28608:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 23:50:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 24 23:50:34 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 24 23:55:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 24 23:55:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 24 23:55:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500965444, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6f600/0xf077f1a829b57326 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a607d57 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 23:55:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 24 23:55:44 oak-gw06 kernel: LustreError: 28624:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88005df08780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 24 23:55:44 oak-gw06 kernel: LustreError: 28624:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 24 23:55:44 oak-gw06 kernel: LustreError: 28624:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88005df08780) refcount = 2 Jul 24 23:55:44 oak-gw06 kernel: LustreError: 28624:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 24 23:55:44 oak-gw06 kernel: LustreError: 28624:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6f600/0xf077f1a829b57326 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a607d57 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 24 23:55:44 oak-gw06 kernel: LustreError: 28624:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 00:00:53 oak-gw06 kernel: LustreError: 28634:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88005df08e40) refcount = 2 Jul 25 00:00:53 oak-gw06 kernel: LustreError: 28634:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 00:00:53 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 00:00:53 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 00:05:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 00:05:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 00:05:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500966059, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6e600/0xf077f1a829b5735e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a617249 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 00:05:59 oak-gw06 kernel: LustreError: 28673:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88005df08240) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 00:05:59 oak-gw06 kernel: LustreError: 28673:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 00:05:59 oak-gw06 kernel: LustreError: 28673:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88005df08240) refcount = 2 Jul 25 00:05:59 oak-gw06 kernel: LustreError: 28673:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 00:05:59 oak-gw06 kernel: LustreError: 28673:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6e600/0xf077f1a829b5735e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a617249 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 00:05:59 oak-gw06 kernel: LustreError: 28673:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 00:05:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 00:11:06 oak-gw06 kernel: LustreError: 28683:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880399dce300) refcount = 2 Jul 25 00:11:06 oak-gw06 kernel: LustreError: 28683:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 00:11:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 00:11:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 00:16:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 00:16:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 00:16:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500966676, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6c000/0xf077f1a829b57396 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a626829 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 00:16:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 00:16:16 oak-gw06 kernel: LustreError: 28686:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800b8ec7900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 00:16:16 oak-gw06 kernel: LustreError: 28686:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 00:16:16 oak-gw06 kernel: LustreError: 28686:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800b8ec7900) refcount = 2 Jul 25 00:16:16 oak-gw06 kernel: LustreError: 28686:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 00:16:16 oak-gw06 kernel: LustreError: 28686:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6c000/0xf077f1a829b57396 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a626829 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 00:16:16 oak-gw06 kernel: LustreError: 28686:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 00:21:22 oak-gw06 kernel: LustreError: 28697:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800b8ec7000) refcount = 2 Jul 25 00:21:22 oak-gw06 kernel: LustreError: 28697:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 00:21:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 00:21:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 00:26:30 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 00:26:30 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 00:26:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500967290, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6f800/0xf077f1a829b573ce lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a63598d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 00:26:30 oak-gw06 kernel: LustreError: 28700:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803606cbd80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 00:26:30 oak-gw06 kernel: LustreError: 28700:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 00:26:30 oak-gw06 kernel: LustreError: 28700:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cbd80) refcount = 2 Jul 25 00:26:30 oak-gw06 kernel: LustreError: 28700:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 00:26:30 oak-gw06 kernel: LustreError: 28700:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6f800/0xf077f1a829b573ce lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a63598d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 00:26:30 oak-gw06 kernel: LustreError: 28700:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 00:26:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 00:31:38 oak-gw06 kernel: LustreError: 28710:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cb900) refcount = 2 Jul 25 00:31:38 oak-gw06 kernel: LustreError: 28710:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 00:31:38 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 00:31:38 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 00:36:47 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 00:36:47 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 00:36:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500967907, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6e400/0xf077f1a829b57406 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a64517a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 00:36:47 oak-gw06 kernel: LustreError: 28713:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803606cb480) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 00:36:47 oak-gw06 kernel: LustreError: 28713:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 00:36:47 oak-gw06 kernel: LustreError: 28713:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cb480) refcount = 2 Jul 25 00:36:47 oak-gw06 kernel: LustreError: 28713:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 00:36:47 oak-gw06 kernel: LustreError: 28713:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6e400/0xf077f1a829b57406 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a64517a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 00:36:47 oak-gw06 kernel: LustreError: 28713:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 00:36:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 00:41:55 oak-gw06 kernel: LustreError: 28724:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cbc00) refcount = 2 Jul 25 00:41:55 oak-gw06 kernel: LustreError: 28724:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 00:41:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 00:41:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 00:47:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 00:47:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 00:47:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500968520, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6c600/0xf077f1a829b5743e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a653e3f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 00:47:00 oak-gw06 kernel: LustreError: 28727:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803606cb180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 00:47:00 oak-gw06 kernel: LustreError: 28727:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 00:47:00 oak-gw06 kernel: LustreError: 28727:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cb180) refcount = 2 Jul 25 00:47:00 oak-gw06 kernel: LustreError: 28727:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 00:47:00 oak-gw06 kernel: LustreError: 28727:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6c600/0xf077f1a829b5743e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a653e3f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 00:47:00 oak-gw06 kernel: LustreError: 28727:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 00:47:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 00:52:09 oak-gw06 kernel: LustreError: 28737:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cbf00) refcount = 2 Jul 25 00:52:09 oak-gw06 kernel: LustreError: 28737:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 00:52:09 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 00:52:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 00:57:15 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 00:57:15 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 00:57:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500969135, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6ea00/0xf077f1a829b57476 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a662f10 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 00:57:15 oak-gw06 kernel: LustreError: 28740:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803606cb900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 00:57:15 oak-gw06 kernel: LustreError: 28740:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 00:57:15 oak-gw06 kernel: LustreError: 28740:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cb900) refcount = 2 Jul 25 00:57:15 oak-gw06 kernel: LustreError: 28740:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 00:57:15 oak-gw06 kernel: LustreError: 28740:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6ea00/0xf077f1a829b57476 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a662f10 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 00:57:15 oak-gw06 kernel: LustreError: 28740:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 00:57:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 01:02:24 oak-gw06 kernel: LustreError: 28784:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cb180) refcount = 2 Jul 25 01:02:24 oak-gw06 kernel: LustreError: 28784:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 01:02:24 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 01:02:24 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 01:07:30 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 01:07:30 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 01:07:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500969750, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6f000/0xf077f1a829b574ae lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a672480 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 01:07:30 oak-gw06 kernel: LustreError: 28787:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880413e69f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 01:07:30 oak-gw06 kernel: LustreError: 28787:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 01:07:30 oak-gw06 kernel: LustreError: 28787:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e69f00) refcount = 2 Jul 25 01:07:30 oak-gw06 kernel: LustreError: 28787:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 01:07:30 oak-gw06 kernel: LustreError: 28787:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6f000/0xf077f1a829b574ae lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a672480 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 01:07:30 oak-gw06 kernel: LustreError: 28787:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 01:07:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 01:12:40 oak-gw06 kernel: LustreError: 28797:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e699c0) refcount = 2 Jul 25 01:12:40 oak-gw06 kernel: LustreError: 28797:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 01:12:40 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 01:12:40 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 01:17:47 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 01:17:47 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 01:17:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500970367, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6e600/0xf077f1a829b574e6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a681b2b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 01:17:47 oak-gw06 kernel: LustreError: 28800:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880413e69f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 01:17:47 oak-gw06 kernel: LustreError: 28800:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 01:17:47 oak-gw06 kernel: LustreError: 28800:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e69f00) refcount = 2 Jul 25 01:17:47 oak-gw06 kernel: LustreError: 28800:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 01:17:47 oak-gw06 kernel: LustreError: 28800:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6e600/0xf077f1a829b574e6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a681b2b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 01:17:47 oak-gw06 kernel: LustreError: 28800:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 01:17:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 01:22:55 oak-gw06 kernel: LustreError: 28810:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e69e40) refcount = 2 Jul 25 01:22:55 oak-gw06 kernel: LustreError: 28810:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 01:22:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 01:22:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 01:28:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 01:28:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 01:28:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500970980, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6d000/0xf077f1a829b5751e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a690a51 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 01:28:00 oak-gw06 kernel: LustreError: 28813:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880413e69240) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 01:28:00 oak-gw06 kernel: LustreError: 28813:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 01:28:00 oak-gw06 kernel: LustreError: 28813:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e69240) refcount = 2 Jul 25 01:28:00 oak-gw06 kernel: LustreError: 28813:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 01:28:00 oak-gw06 kernel: LustreError: 28813:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6d000/0xf077f1a829b5751e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a690a51 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 01:28:00 oak-gw06 kernel: LustreError: 28813:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 01:28:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 01:33:10 oak-gw06 kernel: LustreError: 28824:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e69cc0) refcount = 2 Jul 25 01:33:10 oak-gw06 kernel: LustreError: 28824:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 01:33:10 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 01:33:10 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 01:38:18 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 01:38:18 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 01:38:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500971598, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6f800/0xf077f1a829b57556 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a6a02ca expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 01:38:18 oak-gw06 kernel: LustreError: 28827:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880413e69c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 01:38:18 oak-gw06 kernel: LustreError: 28827:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 01:38:18 oak-gw06 kernel: LustreError: 28827:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e69c00) refcount = 2 Jul 25 01:38:18 oak-gw06 kernel: LustreError: 28827:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 01:38:18 oak-gw06 kernel: LustreError: 28827:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6f800/0xf077f1a829b57556 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a6a02ca expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 01:38:18 oak-gw06 kernel: LustreError: 28827:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 01:38:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 01:43:23 oak-gw06 kernel: LustreError: 28837:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e69540) refcount = 2 Jul 25 01:43:23 oak-gw06 kernel: LustreError: 28837:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 01:43:23 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 01:43:23 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 01:48:30 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 01:48:30 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 01:48:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500972210, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6fc00/0xf077f1a829b5758e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a6af0f4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 01:48:30 oak-gw06 kernel: LustreError: 28841:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880413e69900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 01:48:30 oak-gw06 kernel: LustreError: 28841:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 01:48:30 oak-gw06 kernel: LustreError: 28841:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e69900) refcount = 2 Jul 25 01:48:30 oak-gw06 kernel: LustreError: 28841:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 01:48:30 oak-gw06 kernel: LustreError: 28841:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6fc00/0xf077f1a829b5758e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a6af0f4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 01:48:30 oak-gw06 kernel: LustreError: 28841:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 01:48:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 01:53:38 oak-gw06 kernel: LustreError: 28852:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e69c00) refcount = 2 Jul 25 01:53:38 oak-gw06 kernel: LustreError: 28852:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 01:53:38 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 01:53:38 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 01:58:48 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 01:58:48 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 01:58:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500972828, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6e200/0xf077f1a829b575c6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a6be80f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 01:58:48 oak-gw06 kernel: LustreError: 28855:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880413e69f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 01:58:48 oak-gw06 kernel: LustreError: 28855:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 01:58:48 oak-gw06 kernel: LustreError: 28855:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e69f00) refcount = 2 Jul 25 01:58:48 oak-gw06 kernel: LustreError: 28855:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 01:58:48 oak-gw06 kernel: LustreError: 28855:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6e200/0xf077f1a829b575c6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a6be80f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 01:58:48 oak-gw06 kernel: LustreError: 28855:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 01:58:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 02:03:55 oak-gw06 kernel: LustreError: 28899:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e699c0) refcount = 2 Jul 25 02:03:55 oak-gw06 kernel: LustreError: 28899:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 02:03:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 02:03:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 02:09:05 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 02:09:05 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 02:09:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500973445, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6ea00/0xf077f1a829b575fe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a6cdd9b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 02:09:05 oak-gw06 kernel: LustreError: 28902:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803606cbb40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 02:09:05 oak-gw06 kernel: LustreError: 28902:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 02:09:05 oak-gw06 kernel: LustreError: 28902:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cbb40) refcount = 2 Jul 25 02:09:05 oak-gw06 kernel: LustreError: 28902:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 02:09:05 oak-gw06 kernel: LustreError: 28902:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6ea00/0xf077f1a829b575fe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a6cdd9b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 02:09:05 oak-gw06 kernel: LustreError: 28902:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 02:09:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 02:14:12 oak-gw06 kernel: LustreError: 28912:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cb540) refcount = 2 Jul 25 02:14:12 oak-gw06 kernel: LustreError: 28912:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 02:14:12 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 02:14:12 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 02:19:20 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 02:19:20 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 02:19:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500974060, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6e400/0xf077f1a829b57636 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a6dd0f7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 02:19:20 oak-gw06 kernel: LustreError: 28915:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803606cb900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 02:19:20 oak-gw06 kernel: LustreError: 28915:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 02:19:20 oak-gw06 kernel: LustreError: 28915:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cb900) refcount = 2 Jul 25 02:19:20 oak-gw06 kernel: LustreError: 28915:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 02:19:20 oak-gw06 kernel: LustreError: 28915:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6e400/0xf077f1a829b57636 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a6dd0f7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 02:19:20 oak-gw06 kernel: LustreError: 28915:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 02:19:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 02:24:30 oak-gw06 kernel: LustreError: 28925:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cb9c0) refcount = 2 Jul 25 02:24:30 oak-gw06 kernel: LustreError: 28925:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 02:24:30 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 02:24:30 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 02:29:39 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 02:29:39 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 02:29:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500974679, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6f800/0xf077f1a829b57675 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a6ec923 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 02:29:39 oak-gw06 kernel: LustreError: 28928:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803606cb6c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 02:29:39 oak-gw06 kernel: LustreError: 28928:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 02:29:39 oak-gw06 kernel: LustreError: 28928:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cb6c0) refcount = 2 Jul 25 02:29:39 oak-gw06 kernel: LustreError: 28928:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 02:29:39 oak-gw06 kernel: LustreError: 28928:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6f800/0xf077f1a829b57675 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a6ec923 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 02:29:39 oak-gw06 kernel: LustreError: 28928:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 02:29:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 02:34:49 oak-gw06 kernel: LustreError: 28938:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cba80) refcount = 2 Jul 25 02:34:49 oak-gw06 kernel: LustreError: 28938:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 02:34:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 02:34:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 02:39:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 02:39:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 02:39:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500975297, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6c800/0xf077f1a829b576ad lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a6fc2f3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 02:39:57 oak-gw06 kernel: LustreError: 28941:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803606cb6c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 02:39:57 oak-gw06 kernel: LustreError: 28941:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 02:39:57 oak-gw06 kernel: LustreError: 28941:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cb6c0) refcount = 2 Jul 25 02:39:57 oak-gw06 kernel: LustreError: 28941:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 02:39:57 oak-gw06 kernel: LustreError: 28941:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6c800/0xf077f1a829b576ad lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a6fc2f3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 02:39:57 oak-gw06 kernel: LustreError: 28941:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 02:39:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 02:45:03 oak-gw06 kernel: LustreError: 28951:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cbcc0) refcount = 2 Jul 25 02:45:03 oak-gw06 kernel: LustreError: 28951:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 02:45:03 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 02:45:03 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 02:50:09 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 02:50:09 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 02:50:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500975909, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6f800/0xf077f1a829b576ec lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a70b178 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 02:50:09 oak-gw06 kernel: LustreError: 28961:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803606cb780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 02:50:09 oak-gw06 kernel: LustreError: 28961:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 02:50:09 oak-gw06 kernel: LustreError: 28961:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cb780) refcount = 2 Jul 25 02:50:09 oak-gw06 kernel: LustreError: 28961:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 02:50:09 oak-gw06 kernel: LustreError: 28961:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6f800/0xf077f1a829b576ec lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a70b178 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 02:50:09 oak-gw06 kernel: LustreError: 28961:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 02:50:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 02:55:15 oak-gw06 kernel: LustreError: 28964:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cb540) refcount = 2 Jul 25 02:55:15 oak-gw06 kernel: LustreError: 28964:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 02:55:15 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 02:55:15 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 03:00:25 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 03:00:25 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 03:00:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500976525, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e200/0xf077f1a829b57724 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a71a4d4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 03:00:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 03:00:25 oak-gw06 kernel: LustreError: 28974:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d20c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 03:00:25 oak-gw06 kernel: LustreError: 28974:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 03:00:25 oak-gw06 kernel: LustreError: 28974:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d20c0) refcount = 2 Jul 25 03:00:25 oak-gw06 kernel: LustreError: 28974:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 03:00:25 oak-gw06 kernel: LustreError: 28974:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e200/0xf077f1a829b57724 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a71a4d4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 03:00:25 oak-gw06 kernel: LustreError: 28974:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 03:05:34 oak-gw06 kernel: LustreError: 29008:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2c00) refcount = 2 Jul 25 03:05:34 oak-gw06 kernel: LustreError: 29008:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 03:05:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 03:05:34 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 03:10:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 03:10:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 03:10:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500977141, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ec00/0xf077f1a829b5775c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a729a05 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 03:10:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 03:10:41 oak-gw06 kernel: LustreError: 29016:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 03:10:41 oak-gw06 kernel: LustreError: 29016:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 03:10:41 oak-gw06 kernel: LustreError: 29016:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2900) refcount = 2 Jul 25 03:10:41 oak-gw06 kernel: LustreError: 29016:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 03:10:41 oak-gw06 kernel: LustreError: 29016:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ec00/0xf077f1a829b5775c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a729a05 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 03:10:41 oak-gw06 kernel: LustreError: 29016:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 03:15:47 oak-gw06 kernel: LustreError: 29019:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea240) refcount = 2 Jul 25 03:15:47 oak-gw06 kernel: LustreError: 29019:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 03:15:47 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 03:15:47 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 03:20:54 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 03:20:54 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 03:20:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500977754, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73200/0xf077f1a829b5779b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a738add expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 03:20:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 03:20:54 oak-gw06 kernel: LustreError: 29027:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea480) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 03:20:54 oak-gw06 kernel: LustreError: 29027:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 03:20:54 oak-gw06 kernel: LustreError: 29027:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea480) refcount = 2 Jul 25 03:20:54 oak-gw06 kernel: LustreError: 29027:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 03:20:54 oak-gw06 kernel: LustreError: 29027:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73200/0xf077f1a829b5779b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a738add expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 03:20:54 oak-gw06 kernel: LustreError: 29027:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 03:26:02 oak-gw06 kernel: LustreError: 29056:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880054184c00) refcount = 2 Jul 25 03:26:02 oak-gw06 kernel: LustreError: 29056:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 03:26:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 03:26:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 03:31:08 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 03:31:08 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 03:31:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500978368, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326f800/0xf077f1a829b577d3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a747c80 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 03:31:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 03:31:08 oak-gw06 kernel: LustreError: 29066:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 03:31:08 oak-gw06 kernel: LustreError: 29066:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 03:31:08 oak-gw06 kernel: LustreError: 29066:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2e40) refcount = 2 Jul 25 03:31:08 oak-gw06 kernel: LustreError: 29066:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 03:31:08 oak-gw06 kernel: LustreError: 29066:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326f800/0xf077f1a829b577d3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a747c80 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 03:31:08 oak-gw06 kernel: LustreError: 29066:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 03:36:16 oak-gw06 kernel: LustreError: 29069:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2180) refcount = 2 Jul 25 03:36:16 oak-gw06 kernel: LustreError: 29069:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 03:36:16 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 03:36:16 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 03:41:23 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 03:41:23 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 03:41:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500978983, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326fa00/0xf077f1a829b5780b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a7571aa expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 03:41:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 03:41:23 oak-gw06 kernel: LustreError: 29079:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2240) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 03:41:23 oak-gw06 kernel: LustreError: 29079:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 03:41:23 oak-gw06 kernel: LustreError: 29079:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2240) refcount = 2 Jul 25 03:41:23 oak-gw06 kernel: LustreError: 29079:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 03:41:23 oak-gw06 kernel: LustreError: 29079:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326fa00/0xf077f1a829b5780b lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a7571aa expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 03:41:23 oak-gw06 kernel: LustreError: 29079:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 03:46:32 oak-gw06 kernel: LustreError: 29083:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2c00) refcount = 2 Jul 25 03:46:32 oak-gw06 kernel: LustreError: 29083:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 03:46:32 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 03:46:32 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 03:51:39 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 03:51:39 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 03:51:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500979599, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e400/0xf077f1a829b5784a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a7666fe expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 03:51:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 03:51:39 oak-gw06 kernel: LustreError: 29092:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 03:51:39 oak-gw06 kernel: LustreError: 29092:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 03:51:39 oak-gw06 kernel: LustreError: 29092:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2780) refcount = 2 Jul 25 03:51:39 oak-gw06 kernel: LustreError: 29092:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 03:51:39 oak-gw06 kernel: LustreError: 29092:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e400/0xf077f1a829b5784a lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a7666fe expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 03:51:39 oak-gw06 kernel: LustreError: 29092:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 03:56:46 oak-gw06 kernel: LustreError: 29095:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2900) refcount = 2 Jul 25 03:56:46 oak-gw06 kernel: LustreError: 29095:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 03:56:46 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 03:56:46 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 04:01:56 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 04:01:56 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 04:01:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500980216, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326f200/0xf077f1a829b57889 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a775de8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 04:01:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 04:01:56 oak-gw06 kernel: LustreError: 29137:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 04:01:56 oak-gw06 kernel: LustreError: 29137:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 04:01:56 oak-gw06 kernel: LustreError: 29137:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2cc0) refcount = 2 Jul 25 04:01:56 oak-gw06 kernel: LustreError: 29137:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 04:01:56 oak-gw06 kernel: LustreError: 29137:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326f200/0xf077f1a829b57889 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a775de8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 04:01:56 oak-gw06 kernel: LustreError: 29137:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 04:07:03 oak-gw06 kernel: LustreError: 29140:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b103c0) refcount = 2 Jul 25 04:07:03 oak-gw06 kernel: LustreError: 29140:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 04:07:03 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 04:07:03 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 04:12:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 04:12:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 04:12:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500980833, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0200/0xf077f1a829b578c8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a78536d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 04:12:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 04:12:13 oak-gw06 kernel: LustreError: 29150:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880393a09780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 04:12:13 oak-gw06 kernel: LustreError: 29150:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 04:12:13 oak-gw06 kernel: LustreError: 29150:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880393a09780) refcount = 2 Jul 25 04:12:13 oak-gw06 kernel: LustreError: 29150:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 04:12:13 oak-gw06 kernel: LustreError: 29150:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0200/0xf077f1a829b578c8 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a78536d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 04:12:13 oak-gw06 kernel: LustreError: 29150:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 04:17:23 oak-gw06 kernel: LustreError: 29153:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803606cb6c0) refcount = 2 Jul 25 04:17:23 oak-gw06 kernel: LustreError: 29153:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 04:17:23 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 04:17:23 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 04:22:29 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 04:22:29 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 04:22:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500981449, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56cc00/0xf077f1a829b57907 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a794915 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 04:22:29 oak-gw06 kernel: LustreError: 29163:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 04:22:29 oak-gw06 kernel: LustreError: 29163:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 04:22:29 oak-gw06 kernel: LustreError: 29163:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10780) refcount = 2 Jul 25 04:22:29 oak-gw06 kernel: LustreError: 29163:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 04:22:29 oak-gw06 kernel: LustreError: 29163:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56cc00/0xf077f1a829b57907 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a794915 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 04:22:29 oak-gw06 kernel: LustreError: 29163:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 04:22:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 04:27:35 oak-gw06 kernel: LustreError: 29166:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880167167b40) refcount = 2 Jul 25 04:27:35 oak-gw06 kernel: LustreError: 29166:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 04:27:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 04:27:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 04:32:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 04:32:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 04:32:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500982064, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb3a00/0xf077f1a829b57946 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a7a3c32 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 04:32:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 04:32:44 oak-gw06 kernel: LustreError: 29176:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880167167600) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 04:32:44 oak-gw06 kernel: LustreError: 29176:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 04:32:44 oak-gw06 kernel: LustreError: 29176:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880167167600) refcount = 2 Jul 25 04:32:44 oak-gw06 kernel: LustreError: 29176:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 04:32:44 oak-gw06 kernel: LustreError: 29176:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb3a00/0xf077f1a829b57946 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a7a3c32 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 04:32:44 oak-gw06 kernel: LustreError: 29176:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 04:37:52 oak-gw06 kernel: LustreError: 29179:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880393a09c00) refcount = 2 Jul 25 04:37:52 oak-gw06 kernel: LustreError: 29179:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 04:37:52 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 04:37:52 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 04:42:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 04:42:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 04:42:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500982677, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb3a00/0xf077f1a829b5798c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a7b2d88 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 04:42:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 04:42:57 oak-gw06 kernel: LustreError: 29189:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880393a09cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 04:42:57 oak-gw06 kernel: LustreError: 29189:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 04:42:57 oak-gw06 kernel: LustreError: 29189:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880393a09cc0) refcount = 2 Jul 25 04:42:57 oak-gw06 kernel: LustreError: 29189:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 04:42:57 oak-gw06 kernel: LustreError: 29189:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb3a00/0xf077f1a829b5798c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a7b2d88 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 04:42:57 oak-gw06 kernel: LustreError: 29189:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 04:48:02 oak-gw06 kernel: LustreError: 29192:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880167167180) refcount = 2 Jul 25 04:48:02 oak-gw06 kernel: LustreError: 29192:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 04:48:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 04:48:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 04:53:10 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 04:53:10 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 04:53:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500983290, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b579c4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a7c1d95 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 04:53:10 oak-gw06 kernel: LustreError: 29204:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10300) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 04:53:10 oak-gw06 kernel: LustreError: 29204:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 04:53:10 oak-gw06 kernel: LustreError: 29204:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10300) refcount = 2 Jul 25 04:53:10 oak-gw06 kernel: LustreError: 29204:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 04:53:10 oak-gw06 kernel: LustreError: 29204:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b579c4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a7c1d95 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 04:53:10 oak-gw06 kernel: LustreError: 29204:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 04:53:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 04:58:17 oak-gw06 kernel: LustreError: 29207:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10780) refcount = 2 Jul 25 04:58:17 oak-gw06 kernel: LustreError: 29207:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 04:58:17 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 04:58:17 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 05:03:26 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 05:03:26 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 05:03:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500983906, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b579fc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a7d1375 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 05:03:26 oak-gw06 kernel: LustreError: 29250:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b106c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 05:03:26 oak-gw06 kernel: LustreError: 29250:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 05:03:26 oak-gw06 kernel: LustreError: 29250:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b106c0) refcount = 2 Jul 25 05:03:26 oak-gw06 kernel: LustreError: 29250:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 05:03:26 oak-gw06 kernel: LustreError: 29250:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b579fc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a7d1375 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 05:03:26 oak-gw06 kernel: LustreError: 29250:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 05:03:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 05:08:32 oak-gw06 kernel: LustreError: 29254:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10c00) refcount = 2 Jul 25 05:08:32 oak-gw06 kernel: LustreError: 29254:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 05:08:32 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 05:08:32 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 05:13:39 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 05:13:39 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 05:13:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500984519, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ec00/0xf077f1a829b57a3b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a7e05ab expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 05:13:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 05:13:39 oak-gw06 kernel: LustreError: 29266:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10600) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 05:13:39 oak-gw06 kernel: LustreError: 29266:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 05:13:39 oak-gw06 kernel: LustreError: 29266:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10600) refcount = 2 Jul 25 05:13:39 oak-gw06 kernel: LustreError: 29266:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 05:13:39 oak-gw06 kernel: LustreError: 29266:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ec00/0xf077f1a829b57a3b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a7e05ab expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 05:13:39 oak-gw06 kernel: LustreError: 29266:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 05:18:49 oak-gw06 kernel: LustreError: 29269:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10540) refcount = 2 Jul 25 05:18:49 oak-gw06 kernel: LustreError: 29269:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 05:18:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 05:18:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 05:23:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 05:23:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 05:23:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500985137, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fe00/0xf077f1a829b57a7a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a7efb5a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 05:23:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 05:23:57 oak-gw06 kernel: LustreError: 29281:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 05:23:57 oak-gw06 kernel: LustreError: 29281:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 05:23:57 oak-gw06 kernel: LustreError: 29281:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10b40) refcount = 2 Jul 25 05:23:57 oak-gw06 kernel: LustreError: 29281:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 05:23:57 oak-gw06 kernel: LustreError: 29281:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fe00/0xf077f1a829b57a7a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a7efb5a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 05:23:57 oak-gw06 kernel: LustreError: 29281:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 05:29:04 oak-gw06 kernel: LustreError: 29284:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10c00) refcount = 2 Jul 25 05:29:04 oak-gw06 kernel: LustreError: 29284:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 05:29:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 05:29:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 05:34:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 05:34:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 05:34:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500985751, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b57ab2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a7feca2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 05:34:11 oak-gw06 kernel: LustreError: 29295:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 05:34:11 oak-gw06 kernel: LustreError: 29295:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 05:34:11 oak-gw06 kernel: LustreError: 29295:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10900) refcount = 2 Jul 25 05:34:11 oak-gw06 kernel: LustreError: 29295:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 05:34:11 oak-gw06 kernel: LustreError: 29295:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b57ab2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a7feca2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 05:34:11 oak-gw06 kernel: LustreError: 29295:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 05:34:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 05:39:21 oak-gw06 kernel: LustreError: 29298:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10240) refcount = 2 Jul 25 05:39:21 oak-gw06 kernel: LustreError: 29298:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 05:39:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 05:39:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 05:44:27 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 05:44:27 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 05:44:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500986367, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b57af1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a80e052 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 05:44:27 oak-gw06 kernel: LustreError: 29309:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10600) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 05:44:27 oak-gw06 kernel: LustreError: 29309:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 05:44:27 oak-gw06 kernel: LustreError: 29309:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10600) refcount = 2 Jul 25 05:44:27 oak-gw06 kernel: LustreError: 29309:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 05:44:27 oak-gw06 kernel: LustreError: 29309:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b57af1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a80e052 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 05:44:27 oak-gw06 kernel: LustreError: 29309:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 05:44:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 05:49:37 oak-gw06 kernel: LustreError: 29312:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10180) refcount = 2 Jul 25 05:49:37 oak-gw06 kernel: LustreError: 29312:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 05:49:37 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 05:49:37 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 05:54:45 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 05:54:45 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 05:54:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500986985, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b57b29 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a81d846 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 05:54:45 oak-gw06 kernel: LustreError: 29321:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b109c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 05:54:45 oak-gw06 kernel: LustreError: 29321:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 05:54:45 oak-gw06 kernel: LustreError: 29321:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b109c0) refcount = 2 Jul 25 05:54:45 oak-gw06 kernel: LustreError: 29321:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 05:54:45 oak-gw06 kernel: LustreError: 29321:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b57b29 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a81d846 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 05:54:45 oak-gw06 kernel: LustreError: 29321:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 05:54:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 05:59:52 oak-gw06 kernel: LustreError: 29324:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10540) refcount = 2 Jul 25 05:59:52 oak-gw06 kernel: LustreError: 29324:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 05:59:52 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 05:59:52 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 06:05:01 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 06:05:01 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 06:05:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500987601, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d200/0xf077f1a829b57b61 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a82cdc4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 06:05:01 oak-gw06 kernel: LustreError: 29366:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 06:05:01 oak-gw06 kernel: LustreError: 29366:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 06:05:01 oak-gw06 kernel: LustreError: 29366:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10d80) refcount = 2 Jul 25 06:05:01 oak-gw06 kernel: LustreError: 29366:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 06:05:01 oak-gw06 kernel: LustreError: 29366:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d200/0xf077f1a829b57b61 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a82cdc4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 06:05:01 oak-gw06 kernel: LustreError: 29366:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 06:05:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 06:10:11 oak-gw06 kernel: LustreError: 29376:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10540) refcount = 2 Jul 25 06:10:11 oak-gw06 kernel: LustreError: 29376:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 06:10:11 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 06:10:11 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 06:15:18 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 06:15:18 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 06:15:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500988218, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f400/0xf077f1a829b57b99 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a83c461 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 06:15:18 oak-gw06 kernel: LustreError: 29379:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10600) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 06:15:18 oak-gw06 kernel: LustreError: 29379:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 06:15:18 oak-gw06 kernel: LustreError: 29379:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10600) refcount = 2 Jul 25 06:15:18 oak-gw06 kernel: LustreError: 29379:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 06:15:18 oak-gw06 kernel: LustreError: 29379:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f400/0xf077f1a829b57b99 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a83c461 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 06:15:18 oak-gw06 kernel: LustreError: 29379:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 06:15:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 06:20:26 oak-gw06 kernel: LustreError: 29389:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10300) refcount = 2 Jul 25 06:20:26 oak-gw06 kernel: LustreError: 29389:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 06:20:26 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 06:20:26 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 06:25:35 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 06:25:35 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 06:25:35 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500988835, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ce00/0xf077f1a829b57bd1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a84bba6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 06:25:35 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 06:25:35 oak-gw06 kernel: LustreError: 29392:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 06:25:35 oak-gw06 kernel: LustreError: 29392:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 06:25:35 oak-gw06 kernel: LustreError: 29392:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10c00) refcount = 2 Jul 25 06:25:35 oak-gw06 kernel: LustreError: 29392:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 06:25:35 oak-gw06 kernel: LustreError: 29392:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ce00/0xf077f1a829b57bd1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a84bba6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 06:25:35 oak-gw06 kernel: LustreError: 29392:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 06:30:44 oak-gw06 kernel: LustreError: 29403:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10f00) refcount = 2 Jul 25 06:30:44 oak-gw06 kernel: LustreError: 29403:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 06:30:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 06:30:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 06:35:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 06:35:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 06:35:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500989449, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ec00/0xf077f1a829b57c10 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a85af17 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 06:35:49 oak-gw06 kernel: LustreError: 29406:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10600) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 06:35:49 oak-gw06 kernel: LustreError: 29406:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 06:35:49 oak-gw06 kernel: LustreError: 29406:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10600) refcount = 2 Jul 25 06:35:49 oak-gw06 kernel: LustreError: 29406:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 06:35:49 oak-gw06 kernel: LustreError: 29406:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ec00/0xf077f1a829b57c10 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a85af17 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 06:35:49 oak-gw06 kernel: LustreError: 29406:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 06:35:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 06:40:55 oak-gw06 kernel: LustreError: 29417:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10240) refcount = 2 Jul 25 06:40:55 oak-gw06 kernel: LustreError: 29417:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 06:40:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 06:40:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 06:46:01 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 06:46:01 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 06:46:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500990061, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e600/0xf077f1a829b57c48 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a869e7c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 06:46:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 06:46:01 oak-gw06 kernel: LustreError: 29421:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 06:46:01 oak-gw06 kernel: LustreError: 29421:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 06:46:01 oak-gw06 kernel: LustreError: 29421:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10000) refcount = 2 Jul 25 06:46:01 oak-gw06 kernel: LustreError: 29421:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 06:46:01 oak-gw06 kernel: LustreError: 29421:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e600/0xf077f1a829b57c48 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a869e7c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 06:46:01 oak-gw06 kernel: LustreError: 29421:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 06:51:06 oak-gw06 kernel: LustreError: 29432:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b103c0) refcount = 2 Jul 25 06:51:06 oak-gw06 kernel: LustreError: 29432:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 06:51:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 06:51:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 06:56:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 06:56:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 06:56:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500990673, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b57c80 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a878eba expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 06:56:13 oak-gw06 kernel: LustreError: 29435:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10840) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 06:56:13 oak-gw06 kernel: LustreError: 29435:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 06:56:13 oak-gw06 kernel: LustreError: 29435:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10840) refcount = 2 Jul 25 06:56:13 oak-gw06 kernel: LustreError: 29435:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 06:56:13 oak-gw06 kernel: LustreError: 29435:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b57c80 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a878eba expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 06:56:13 oak-gw06 kernel: LustreError: 29435:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 06:56:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 07:01:20 oak-gw06 kernel: LustreError: 29478:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3b10480) refcount = 2 Jul 25 07:01:20 oak-gw06 kernel: LustreError: 29478:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 07:01:20 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 07:01:20 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 07:06:30 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 07:06:30 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 07:06:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500991290, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0e00/0xf077f1a829b57cb8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a88843f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 07:06:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 07:06:30 oak-gw06 kernel: LustreError: 29481:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880167167180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 07:06:30 oak-gw06 kernel: LustreError: 29481:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 07:06:30 oak-gw06 kernel: LustreError: 29481:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880167167180) refcount = 2 Jul 25 07:06:30 oak-gw06 kernel: LustreError: 29481:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 07:06:30 oak-gw06 kernel: LustreError: 29481:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0e00/0xf077f1a829b57cb8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a88843f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 07:06:30 oak-gw06 kernel: LustreError: 29481:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 07:11:36 oak-gw06 kernel: LustreError: 29491:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801671670c0) refcount = 2 Jul 25 07:11:36 oak-gw06 kernel: LustreError: 29491:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 07:11:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 07:11:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 07:16:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 07:16:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 07:16:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500991901, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb3400/0xf077f1a829b57cf0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a89713c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 07:16:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 07:16:41 oak-gw06 kernel: LustreError: 29494:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801671679c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 07:16:41 oak-gw06 kernel: LustreError: 29494:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 07:16:41 oak-gw06 kernel: LustreError: 29494:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801671679c0) refcount = 2 Jul 25 07:16:41 oak-gw06 kernel: LustreError: 29494:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 07:16:41 oak-gw06 kernel: LustreError: 29494:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb3400/0xf077f1a829b57cf0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a89713c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 07:16:41 oak-gw06 kernel: LustreError: 29494:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 07:21:47 oak-gw06 kernel: LustreError: 29504:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801671673c0) refcount = 2 Jul 25 07:21:47 oak-gw06 kernel: LustreError: 29504:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 07:21:47 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 07:21:47 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 07:26:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 07:26:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 07:26:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500992517, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ee00/0xf077f1a829b57d28 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a8a6586 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 07:26:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 07:26:57 oak-gw06 kernel: LustreError: 29507:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 07:26:57 oak-gw06 kernel: LustreError: 29507:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 07:26:57 oak-gw06 kernel: LustreError: 29507:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6a80) refcount = 2 Jul 25 07:26:57 oak-gw06 kernel: LustreError: 29507:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 07:26:57 oak-gw06 kernel: LustreError: 29507:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ee00/0xf077f1a829b57d28 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a8a6586 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 07:26:57 oak-gw06 kernel: LustreError: 29507:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 07:32:05 oak-gw06 kernel: LustreError: 29517:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6b40) refcount = 2 Jul 25 07:32:05 oak-gw06 kernel: LustreError: 29517:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 07:32:05 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 07:32:05 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 07:37:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 07:37:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 07:37:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500993134, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c600/0xf077f1a829b57d60 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a8b5c54 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 07:37:14 oak-gw06 kernel: LustreError: 29520:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 07:37:14 oak-gw06 kernel: LustreError: 29520:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 07:37:14 oak-gw06 kernel: LustreError: 29520:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6cc0) refcount = 2 Jul 25 07:37:14 oak-gw06 kernel: LustreError: 29520:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 07:37:14 oak-gw06 kernel: LustreError: 29520:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c600/0xf077f1a829b57d60 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a8b5c54 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 07:37:14 oak-gw06 kernel: LustreError: 29520:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 07:37:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 07:42:22 oak-gw06 kernel: LustreError: 29531:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6f00) refcount = 2 Jul 25 07:42:22 oak-gw06 kernel: LustreError: 29531:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 07:42:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 07:42:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 07:47:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 07:47:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 07:47:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500993751, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b57d98 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a8c5314 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 07:47:31 oak-gw06 kernel: LustreError: 29534:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6600) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 07:47:31 oak-gw06 kernel: LustreError: 29534:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 07:47:31 oak-gw06 kernel: LustreError: 29534:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6600) refcount = 2 Jul 25 07:47:31 oak-gw06 kernel: LustreError: 29534:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 07:47:31 oak-gw06 kernel: LustreError: 29534:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b57d98 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a8c5314 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 07:47:31 oak-gw06 kernel: LustreError: 29534:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 07:47:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 07:52:40 oak-gw06 kernel: LustreError: 29545:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e69c0) refcount = 2 Jul 25 07:52:40 oak-gw06 kernel: LustreError: 29545:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 07:52:40 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 07:52:40 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 07:57:46 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 07:57:46 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 07:57:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500994366, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c000/0xf077f1a829b57dd0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a8d4757 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 07:57:46 oak-gw06 kernel: LustreError: 29548:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 07:57:46 oak-gw06 kernel: LustreError: 29548:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 07:57:46 oak-gw06 kernel: LustreError: 29548:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6180) refcount = 2 Jul 25 07:57:46 oak-gw06 kernel: LustreError: 29548:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 07:57:46 oak-gw06 kernel: LustreError: 29548:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c000/0xf077f1a829b57dd0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a8d4757 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 07:57:46 oak-gw06 kernel: LustreError: 29548:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 07:57:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 08:02:52 oak-gw06 kernel: LustreError: 29590:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6900) refcount = 2 Jul 25 08:02:52 oak-gw06 kernel: LustreError: 29590:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 08:02:52 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 08:02:52 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 08:07:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 08:07:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 08:07:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500994977, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56dc00/0xf077f1a829b57e08 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a8e3518 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 08:07:57 oak-gw06 kernel: LustreError: 29593:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800b8ec70c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 08:07:57 oak-gw06 kernel: LustreError: 29593:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 08:07:57 oak-gw06 kernel: LustreError: 29593:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800b8ec70c0) refcount = 2 Jul 25 08:07:57 oak-gw06 kernel: LustreError: 29593:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 08:07:57 oak-gw06 kernel: LustreError: 29593:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56dc00/0xf077f1a829b57e08 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a8e3518 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 08:07:57 oak-gw06 kernel: LustreError: 29593:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 08:07:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 08:13:04 oak-gw06 kernel: LustreError: 29605:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800b8ec7cc0) refcount = 2 Jul 25 08:13:04 oak-gw06 kernel: LustreError: 29605:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 08:13:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 08:13:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 08:18:12 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 08:18:12 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 08:18:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500995592, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ec00/0xf077f1a829b57e40 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a8f286d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 08:18:12 oak-gw06 kernel: LustreError: 29608:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800b8ec7f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 08:18:12 oak-gw06 kernel: LustreError: 29608:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 08:18:12 oak-gw06 kernel: LustreError: 29608:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800b8ec7f00) refcount = 2 Jul 25 08:18:12 oak-gw06 kernel: LustreError: 29608:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 08:18:12 oak-gw06 kernel: LustreError: 29608:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ec00/0xf077f1a829b57e40 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a8f286d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 08:18:12 oak-gw06 kernel: LustreError: 29608:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 08:18:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 08:23:22 oak-gw06 kernel: LustreError: 29618:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800b8ec7000) refcount = 2 Jul 25 08:23:22 oak-gw06 kernel: LustreError: 29618:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 08:23:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 08:23:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 08:28:29 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 08:28:29 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 08:28:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500996209, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f600/0xf077f1a829b57e78 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a901ed9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 08:28:29 oak-gw06 kernel: LustreError: 29622:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6300) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 08:28:29 oak-gw06 kernel: LustreError: 29622:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 08:28:29 oak-gw06 kernel: LustreError: 29622:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6300) refcount = 2 Jul 25 08:28:29 oak-gw06 kernel: LustreError: 29622:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 08:28:29 oak-gw06 kernel: LustreError: 29622:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f600/0xf077f1a829b57e78 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a901ed9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 08:28:29 oak-gw06 kernel: LustreError: 29622:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 08:28:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 08:33:39 oak-gw06 kernel: LustreError: 29632:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e60c0) refcount = 2 Jul 25 08:33:39 oak-gw06 kernel: LustreError: 29632:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 08:33:39 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 08:33:39 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 08:38:46 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 08:38:46 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 08:38:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500996826, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e200/0xf077f1a829b57eb0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a9112e4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 08:38:46 oak-gw06 kernel: LustreError: 29635:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6480) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 08:38:46 oak-gw06 kernel: LustreError: 29635:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 08:38:46 oak-gw06 kernel: LustreError: 29635:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6480) refcount = 2 Jul 25 08:38:46 oak-gw06 kernel: LustreError: 29635:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 08:38:46 oak-gw06 kernel: LustreError: 29635:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e200/0xf077f1a829b57eb0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a9112e4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 08:38:46 oak-gw06 kernel: LustreError: 29635:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 08:38:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 08:43:54 oak-gw06 kernel: LustreError: 29645:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6540) refcount = 2 Jul 25 08:43:54 oak-gw06 kernel: LustreError: 29645:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 08:43:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 08:43:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 08:49:01 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 08:49:01 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 08:49:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500997441, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b57eef lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a92043a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 08:49:01 oak-gw06 kernel: LustreError: 29648:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e66c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 08:49:01 oak-gw06 kernel: LustreError: 29648:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 08:49:01 oak-gw06 kernel: LustreError: 29648:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e66c0) refcount = 2 Jul 25 08:49:01 oak-gw06 kernel: LustreError: 29648:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 08:49:01 oak-gw06 kernel: LustreError: 29648:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b57eef lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a92043a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 08:49:01 oak-gw06 kernel: LustreError: 29648:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 08:49:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 08:54:09 oak-gw06 kernel: LustreError: 29659:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6f00) refcount = 2 Jul 25 08:54:09 oak-gw06 kernel: LustreError: 29659:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 08:54:09 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 08:54:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 08:59:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 08:59:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 08:59:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500998054, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e800/0xf077f1a829b57f2e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a92f56d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 08:59:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 08:59:14 oak-gw06 kernel: LustreError: 29662:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 08:59:14 oak-gw06 kernel: LustreError: 29662:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 08:59:14 oak-gw06 kernel: LustreError: 29662:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6e40) refcount = 2 Jul 25 08:59:14 oak-gw06 kernel: LustreError: 29662:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 08:59:14 oak-gw06 kernel: LustreError: 29662:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e800/0xf077f1a829b57f2e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a92f56d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 08:59:14 oak-gw06 kernel: LustreError: 29662:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 09:04:22 oak-gw06 kernel: LustreError: 29705:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6d80) refcount = 2 Jul 25 09:04:22 oak-gw06 kernel: LustreError: 29705:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 09:04:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 09:04:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 09:09:29 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 09:09:29 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 09:09:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500998669, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e200/0xf077f1a829b57f6d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a93e8fa expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 09:09:29 oak-gw06 kernel: LustreError: 29708:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 09:09:29 oak-gw06 kernel: LustreError: 29708:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 09:09:29 oak-gw06 kernel: LustreError: 29708:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e6900) refcount = 2 Jul 25 09:09:29 oak-gw06 kernel: LustreError: 29708:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 09:09:29 oak-gw06 kernel: LustreError: 29708:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e200/0xf077f1a829b57f6d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a93e8fa expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 09:09:29 oak-gw06 kernel: LustreError: 29708:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 09:09:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 09:10:44 oak-gw06 kernel: Lustre: oak-OST0002-osc-ffff88041b99c000: Connection to oak-OST0002 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jul 25 09:10:44 oak-gw06 kernel: LustreError: 11-0: oak-OST0012-osc-ffff88041b99c000: operation obd_ping to node 10.0.2.102@o2ib5 failed: rc = -107 Jul 25 09:10:44 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Jul 25 09:10:44 oak-gw06 kernel: Lustre: Skipped 31 previous similar messages Jul 25 09:11:09 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1500999069/real 1500999069] req@ffff8800a0565800 x1566265998740272/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1500999075 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jul 25 09:11:09 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 25 09:11:59 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1500999119/real 1500999119] req@ffff880195b54c00 x1566265998741648/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1500999130 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jul 25 09:11:59 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Jul 25 09:13:30 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1500999194/real 1500999194] req@ffff8802076bb000 x1566265998743920/t0(0) o8->oak-OST000a-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1500999210 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 25 09:13:30 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 39 previous similar messages Jul 25 09:14:35 oak-gw06 kernel: LustreError: 29778:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ed2e66c0) refcount = 2 Jul 25 09:14:35 oak-gw06 kernel: LustreError: 29778:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 09:14:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 09:14:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 09:19:40 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 09:19:40 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 09:19:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500999280, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0000/0xf077f1a829b57fa5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a94d9fc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 09:19:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 09:19:40 oak-gw06 kernel: LustreError: 29801:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ec384480) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 09:19:40 oak-gw06 kernel: LustreError: 29801:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 09:19:40 oak-gw06 kernel: LustreError: 29801:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384480) refcount = 2 Jul 25 09:19:40 oak-gw06 kernel: LustreError: 29801:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 09:19:40 oak-gw06 kernel: LustreError: 29801:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0000/0xf077f1a829b57fa5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a94d9fc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 09:19:40 oak-gw06 kernel: LustreError: 29801:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 09:24:48 oak-gw06 kernel: LustreError: 29821:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384180) refcount = 2 Jul 25 09:24:48 oak-gw06 kernel: LustreError: 29821:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 09:24:48 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 09:24:48 oak-gw06 kernel: Lustre: Skipped 22 previous similar messages Jul 25 09:29:54 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 09:29:54 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 09:29:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1500999894, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2c00/0xf077f1a829b57fdd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a95cabf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 09:29:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 09:29:54 oak-gw06 kernel: LustreError: 29824:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880167167a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 09:29:54 oak-gw06 kernel: LustreError: 29824:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 09:29:54 oak-gw06 kernel: LustreError: 29824:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880167167a80) refcount = 2 Jul 25 09:29:54 oak-gw06 kernel: LustreError: 29824:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 09:29:54 oak-gw06 kernel: LustreError: 29824:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2c00/0xf077f1a829b57fdd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a95cabf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 09:29:54 oak-gw06 kernel: LustreError: 29824:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 09:35:02 oak-gw06 kernel: LustreError: 29835:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384cc0) refcount = 2 Jul 25 09:35:02 oak-gw06 kernel: LustreError: 29835:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 09:35:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 09:35:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 09:40:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 09:40:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 09:40:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501000513, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1400/0xf077f1a829b58015 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a96c1e1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 09:40:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 09:40:13 oak-gw06 kernel: LustreError: 29845:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ec3849c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 09:40:13 oak-gw06 kernel: LustreError: 29845:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 09:40:13 oak-gw06 kernel: LustreError: 29845:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec3849c0) refcount = 2 Jul 25 09:40:13 oak-gw06 kernel: LustreError: 29845:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 09:40:13 oak-gw06 kernel: LustreError: 29845:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1400/0xf077f1a829b58015 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a96c1e1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 09:40:13 oak-gw06 kernel: LustreError: 29845:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 09:45:19 oak-gw06 kernel: LustreError: 29848:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801671679c0) refcount = 2 Jul 25 09:45:19 oak-gw06 kernel: LustreError: 29848:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 09:45:19 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 09:45:19 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 09:50:27 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 09:50:27 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 09:50:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501001127, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2e00/0xf077f1a829b58054 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a97b34c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 09:50:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 09:50:27 oak-gw06 kernel: LustreError: 29858:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801671673c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 09:50:27 oak-gw06 kernel: LustreError: 29858:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 09:50:27 oak-gw06 kernel: LustreError: 29858:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801671673c0) refcount = 2 Jul 25 09:50:27 oak-gw06 kernel: LustreError: 29858:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 09:50:27 oak-gw06 kernel: LustreError: 29858:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2e00/0xf077f1a829b58054 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a97b34c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 09:50:27 oak-gw06 kernel: LustreError: 29858:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 09:55:35 oak-gw06 kernel: LustreError: 29861:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec3843c0) refcount = 2 Jul 25 09:55:35 oak-gw06 kernel: LustreError: 29861:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 09:55:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 09:55:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 10:00:43 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 10:00:43 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 10:00:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501001743, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0200/0xf077f1a829b5809a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a98a78f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 10:00:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 10:00:43 oak-gw06 kernel: LustreError: 29881:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ec384000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 10:00:43 oak-gw06 kernel: LustreError: 29881:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 10:00:43 oak-gw06 kernel: LustreError: 29881:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384000) refcount = 2 Jul 25 10:00:43 oak-gw06 kernel: LustreError: 29881:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 10:00:43 oak-gw06 kernel: LustreError: 29881:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0200/0xf077f1a829b5809a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a98a78f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 10:00:43 oak-gw06 kernel: LustreError: 29881:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 10:05:50 oak-gw06 kernel: LustreError: 29918:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880167167180) refcount = 2 Jul 25 10:05:50 oak-gw06 kernel: LustreError: 29918:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 10:05:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 10:05:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 10:11:01 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 10:11:01 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 10:11:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501002361, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0200/0xf077f1a829b580d9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a999cf1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 10:11:01 oak-gw06 kernel: LustreError: 29928:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880167167d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 10:11:01 oak-gw06 kernel: LustreError: 29928:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 10:11:01 oak-gw06 kernel: LustreError: 29928:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880167167d80) refcount = 2 Jul 25 10:11:01 oak-gw06 kernel: LustreError: 29928:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 10:11:01 oak-gw06 kernel: LustreError: 29928:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0200/0xf077f1a829b580d9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a999cf1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 10:11:01 oak-gw06 kernel: LustreError: 29928:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 10:11:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 10:16:06 oak-gw06 kernel: LustreError: 29931:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384c00) refcount = 2 Jul 25 10:16:06 oak-gw06 kernel: LustreError: 29931:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 10:16:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 10:16:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 10:21:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 10:21:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 10:21:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501002974, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0200/0xf077f1a829b58111 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a9a8e9b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 10:21:14 oak-gw06 kernel: LustreError: 29962:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ec384780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 10:21:14 oak-gw06 kernel: LustreError: 29962:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 10:21:14 oak-gw06 kernel: LustreError: 29962:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384780) refcount = 2 Jul 25 10:21:14 oak-gw06 kernel: LustreError: 29962:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 10:21:14 oak-gw06 kernel: LustreError: 29962:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0200/0xf077f1a829b58111 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a9a8e9b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 10:21:14 oak-gw06 kernel: LustreError: 29962:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 10:21:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 10:26:24 oak-gw06 kernel: LustreError: 29965:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880167167240) refcount = 2 Jul 25 10:26:24 oak-gw06 kernel: LustreError: 29965:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 10:26:24 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 10:26:24 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 10:31:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 10:31:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 10:31:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501003591, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0200/0xf077f1a829b58149 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a9b84c8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 10:31:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 10:31:31 oak-gw06 kernel: LustreError: 29975:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880167167180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 10:31:31 oak-gw06 kernel: LustreError: 29975:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 10:31:31 oak-gw06 kernel: LustreError: 29975:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880167167180) refcount = 2 Jul 25 10:31:31 oak-gw06 kernel: LustreError: 29975:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 10:31:31 oak-gw06 kernel: LustreError: 29975:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0200/0xf077f1a829b58149 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a9b84c8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 10:31:31 oak-gw06 kernel: LustreError: 29975:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 10:36:39 oak-gw06 kernel: LustreError: 29978:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880167167b40) refcount = 2 Jul 25 10:36:39 oak-gw06 kernel: LustreError: 29978:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 10:36:39 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 10:36:39 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 10:41:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 10:41:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 10:41:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501004209, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f600/0xf077f1a829b58181 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a9c7a31 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 10:41:49 oak-gw06 kernel: LustreError: 29988:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88006904f600) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 10:41:49 oak-gw06 kernel: LustreError: 29988:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 10:41:49 oak-gw06 kernel: LustreError: 29988:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88006904f600) refcount = 2 Jul 25 10:41:49 oak-gw06 kernel: LustreError: 29988:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 10:41:49 oak-gw06 kernel: LustreError: 29988:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f600/0xf077f1a829b58181 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a9c7a31 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 10:41:49 oak-gw06 kernel: LustreError: 29988:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 10:41:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 10:46:59 oak-gw06 kernel: LustreError: 29992:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88006904ff00) refcount = 2 Jul 25 10:46:59 oak-gw06 kernel: LustreError: 29992:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 10:46:59 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 10:46:59 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 10:52:08 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 10:52:08 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 10:52:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501004828, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fe00/0xf077f1a829b581b9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a9d7209 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 10:52:08 oak-gw06 kernel: LustreError: 30002:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88006904f6c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 10:52:08 oak-gw06 kernel: LustreError: 30002:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 10:52:08 oak-gw06 kernel: LustreError: 30002:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88006904f6c0) refcount = 2 Jul 25 10:52:08 oak-gw06 kernel: LustreError: 30002:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 10:52:08 oak-gw06 kernel: LustreError: 30002:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fe00/0xf077f1a829b581b9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a9d7209 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 10:52:08 oak-gw06 kernel: LustreError: 30002:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 10:52:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 10:57:17 oak-gw06 kernel: LustreError: 30005:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88006904f900) refcount = 2 Jul 25 10:57:17 oak-gw06 kernel: LustreError: 30005:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 10:57:17 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 10:57:17 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 11:02:24 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 11:02:24 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 11:02:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501005444, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ce00/0xf077f1a829b581f1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a9e6511 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 11:02:24 oak-gw06 kernel: LustreError: 30048:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88006904f000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 11:02:24 oak-gw06 kernel: LustreError: 30048:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 11:02:24 oak-gw06 kernel: LustreError: 30048:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88006904f000) refcount = 2 Jul 25 11:02:24 oak-gw06 kernel: LustreError: 30048:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 11:02:24 oak-gw06 kernel: LustreError: 30048:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ce00/0xf077f1a829b581f1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a9e6511 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 11:02:24 oak-gw06 kernel: LustreError: 30048:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 11:02:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 11:07:32 oak-gw06 kernel: LustreError: 30051:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88006904fe40) refcount = 2 Jul 25 11:07:32 oak-gw06 kernel: LustreError: 30051:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 11:07:32 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 11:07:32 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 11:12:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 11:12:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 11:12:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501006062, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fa00/0xf077f1a829b58229 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6a9f5b61 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 11:12:42 oak-gw06 kernel: LustreError: 30061:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88006904f6c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 11:12:42 oak-gw06 kernel: LustreError: 30061:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 11:12:42 oak-gw06 kernel: LustreError: 30061:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88006904f6c0) refcount = 2 Jul 25 11:12:42 oak-gw06 kernel: LustreError: 30061:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 11:12:42 oak-gw06 kernel: LustreError: 30061:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fa00/0xf077f1a829b58229 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6a9f5b61 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 11:12:42 oak-gw06 kernel: LustreError: 30061:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 11:12:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 11:17:49 oak-gw06 kernel: LustreError: 30064:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88006904f840) refcount = 2 Jul 25 11:17:49 oak-gw06 kernel: LustreError: 30064:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 11:17:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 11:17:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 11:22:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 11:22:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 11:22:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501006679, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ee00/0xf077f1a829b58261 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aa04fc7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 11:22:59 oak-gw06 kernel: LustreError: 30075:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88006904f900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 11:22:59 oak-gw06 kernel: LustreError: 30075:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 11:22:59 oak-gw06 kernel: LustreError: 30075:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88006904f900) refcount = 2 Jul 25 11:22:59 oak-gw06 kernel: LustreError: 30075:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 11:22:59 oak-gw06 kernel: LustreError: 30075:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ee00/0xf077f1a829b58261 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aa04fc7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 11:22:59 oak-gw06 kernel: LustreError: 30075:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 11:22:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 11:28:06 oak-gw06 kernel: LustreError: 30078:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88006904fe40) refcount = 2 Jul 25 11:28:06 oak-gw06 kernel: LustreError: 30078:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 11:28:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 11:28:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 11:33:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 11:33:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 11:33:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501007293, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b58299 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aa1419b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 11:33:13 oak-gw06 kernel: LustreError: 30089:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88006904f480) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 11:33:13 oak-gw06 kernel: LustreError: 30089:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 11:33:13 oak-gw06 kernel: LustreError: 30089:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88006904f480) refcount = 2 Jul 25 11:33:13 oak-gw06 kernel: LustreError: 30089:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 11:33:13 oak-gw06 kernel: LustreError: 30089:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d400/0xf077f1a829b58299 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aa1419b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 11:33:13 oak-gw06 kernel: LustreError: 30089:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 11:33:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 11:38:19 oak-gw06 kernel: LustreError: 30092:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88006904f900) refcount = 2 Jul 25 11:38:19 oak-gw06 kernel: LustreError: 30092:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 11:38:19 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 11:38:19 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 11:43:27 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 11:43:27 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 11:43:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501007907, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ca00/0xf077f1a829b582d1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aa2345d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 11:43:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 11:43:27 oak-gw06 kernel: LustreError: 30102:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 11:43:27 oak-gw06 kernel: LustreError: 30102:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 11:43:27 oak-gw06 kernel: LustreError: 30102:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2900) refcount = 2 Jul 25 11:43:27 oak-gw06 kernel: LustreError: 30102:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 11:43:27 oak-gw06 kernel: LustreError: 30102:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ca00/0xf077f1a829b582d1 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aa2345d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 11:43:27 oak-gw06 kernel: LustreError: 30102:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 11:48:34 oak-gw06 kernel: LustreError: 30106:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2300) refcount = 2 Jul 25 11:48:34 oak-gw06 kernel: LustreError: 30106:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 11:48:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 11:48:34 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 11:53:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 11:53:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 11:53:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501008521, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e000/0xf077f1a829b58317 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aa326af expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 11:53:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 11:53:41 oak-gw06 kernel: LustreError: 30116:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d20c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 11:53:41 oak-gw06 kernel: LustreError: 30116:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 11:53:41 oak-gw06 kernel: LustreError: 30116:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d20c0) refcount = 2 Jul 25 11:53:41 oak-gw06 kernel: LustreError: 30116:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 11:53:41 oak-gw06 kernel: LustreError: 30116:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e000/0xf077f1a829b58317 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aa326af expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 11:53:41 oak-gw06 kernel: LustreError: 30116:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 11:58:51 oak-gw06 kernel: LustreError: 30119:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2d80) refcount = 2 Jul 25 11:58:51 oak-gw06 kernel: LustreError: 30119:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 11:58:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 11:58:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 12:03:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 12:03:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 12:03:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501009139, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326f000/0xf077f1a829b58356 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aa41e72 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 12:03:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 12:03:59 oak-gw06 kernel: LustreError: 30161:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 12:03:59 oak-gw06 kernel: LustreError: 30161:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 12:03:59 oak-gw06 kernel: LustreError: 30161:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2900) refcount = 2 Jul 25 12:03:59 oak-gw06 kernel: LustreError: 30161:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 12:03:59 oak-gw06 kernel: LustreError: 30161:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326f000/0xf077f1a829b58356 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aa41e72 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 12:03:59 oak-gw06 kernel: LustreError: 30161:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 12:09:08 oak-gw06 kernel: LustreError: 30164:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2900) refcount = 2 Jul 25 12:09:08 oak-gw06 kernel: LustreError: 30164:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 12:09:08 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 12:09:08 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 12:14:17 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 12:14:17 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 12:14:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501009757, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326d800/0xf077f1a829b5838e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aa515a9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 12:14:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 12:14:17 oak-gw06 kernel: LustreError: 30175:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 12:14:17 oak-gw06 kernel: LustreError: 30175:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 12:14:17 oak-gw06 kernel: LustreError: 30175:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2900) refcount = 2 Jul 25 12:14:17 oak-gw06 kernel: LustreError: 30175:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 12:14:17 oak-gw06 kernel: LustreError: 30175:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326d800/0xf077f1a829b5838e lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aa515a9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 12:14:17 oak-gw06 kernel: LustreError: 30175:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 12:19:26 oak-gw06 kernel: LustreError: 30178:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2480) refcount = 2 Jul 25 12:19:26 oak-gw06 kernel: LustreError: 30178:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 12:19:26 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 12:19:26 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 12:24:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 12:24:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 12:24:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501010371, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326fe00/0xf077f1a829b583d4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aa605f5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 12:24:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 12:24:31 oak-gw06 kernel: LustreError: 30188:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2540) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 12:24:31 oak-gw06 kernel: LustreError: 30188:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 12:24:31 oak-gw06 kernel: LustreError: 30188:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2540) refcount = 2 Jul 25 12:24:31 oak-gw06 kernel: LustreError: 30188:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 12:24:31 oak-gw06 kernel: LustreError: 30188:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326fe00/0xf077f1a829b583d4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aa605f5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 12:24:31 oak-gw06 kernel: LustreError: 30188:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 12:29:41 oak-gw06 kernel: LustreError: 30191:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2300) refcount = 2 Jul 25 12:29:41 oak-gw06 kernel: LustreError: 30191:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 12:29:41 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 12:29:41 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 12:34:50 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 12:34:50 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 12:34:50 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501010990, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e400/0xf077f1a829b5840c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aa6fd10 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 12:34:50 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 12:34:50 oak-gw06 kernel: LustreError: 30201:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d23c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 12:34:50 oak-gw06 kernel: LustreError: 30201:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 12:34:50 oak-gw06 kernel: LustreError: 30201:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d23c0) refcount = 2 Jul 25 12:34:50 oak-gw06 kernel: LustreError: 30201:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 12:34:50 oak-gw06 kernel: LustreError: 30201:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e400/0xf077f1a829b5840c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aa6fd10 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 12:34:50 oak-gw06 kernel: LustreError: 30201:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 12:40:00 oak-gw06 kernel: LustreError: 30204:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2d80) refcount = 2 Jul 25 12:40:00 oak-gw06 kernel: LustreError: 30204:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 12:40:00 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 12:40:00 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 12:45:05 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 12:45:05 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 12:45:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501011605, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326d400/0xf077f1a829b58444 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aa7ef70 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 12:45:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 12:45:05 oak-gw06 kernel: LustreError: 30215:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d29c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 12:45:05 oak-gw06 kernel: LustreError: 30215:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 12:45:05 oak-gw06 kernel: LustreError: 30215:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d29c0) refcount = 2 Jul 25 12:45:05 oak-gw06 kernel: LustreError: 30215:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 12:45:05 oak-gw06 kernel: LustreError: 30215:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326d400/0xf077f1a829b58444 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aa7ef70 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 12:45:05 oak-gw06 kernel: LustreError: 30215:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 12:50:13 oak-gw06 kernel: LustreError: 30226:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2e40) refcount = 2 Jul 25 12:50:13 oak-gw06 kernel: LustreError: 30226:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 12:50:13 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 12:50:13 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 12:55:19 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 12:55:19 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 12:55:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501012219, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326dc00/0xf077f1a829b58483 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aa8e14b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 12:55:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 12:55:19 oak-gw06 kernel: LustreError: 30229:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d20c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 12:55:19 oak-gw06 kernel: LustreError: 30229:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 12:55:19 oak-gw06 kernel: LustreError: 30229:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d20c0) refcount = 2 Jul 25 12:55:19 oak-gw06 kernel: LustreError: 30229:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 12:55:19 oak-gw06 kernel: LustreError: 30229:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326dc00/0xf077f1a829b58483 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aa8e14b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 12:55:19 oak-gw06 kernel: LustreError: 30229:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 13:00:26 oak-gw06 kernel: LustreError: 30240:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2c00) refcount = 2 Jul 25 13:00:26 oak-gw06 kernel: LustreError: 30240:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 13:00:26 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 13:00:26 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 13:05:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 13:05:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 13:05:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501012831, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326d800/0xf077f1a829b584c9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aa9d0cc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 13:05:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 13:05:31 oak-gw06 kernel: LustreError: 30275:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 13:05:31 oak-gw06 kernel: LustreError: 30275:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 13:05:31 oak-gw06 kernel: LustreError: 30275:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2780) refcount = 2 Jul 25 13:05:31 oak-gw06 kernel: LustreError: 30275:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 13:05:31 oak-gw06 kernel: LustreError: 30275:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326d800/0xf077f1a829b584c9 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aa9d0cc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 13:05:31 oak-gw06 kernel: LustreError: 30275:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 13:10:38 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 13:10:38 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 13:15:47 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 13:15:47 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 13:15:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501013447, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ee00/0xf077f1a829b584f3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aaac49f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 13:15:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 13:15:47 oak-gw06 kernel: LustreError: 30287:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d26c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 13:15:47 oak-gw06 kernel: LustreError: 30287:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d26c0) refcount = 2 Jul 25 13:15:47 oak-gw06 kernel: LustreError: 30287:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 13:15:47 oak-gw06 kernel: LustreError: 30287:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ee00/0xf077f1a829b584f3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aaac49f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 13:20:56 oak-gw06 kernel: LustreError: 30297:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2300) refcount = 2 Jul 25 13:20:56 oak-gw06 kernel: LustreError: 30297:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 13:20:56 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 13:20:56 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 13:26:03 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 13:26:03 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 13:26:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501014063, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ee00/0xf077f1a829b58532 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aabb8bf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 13:26:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 13:26:03 oak-gw06 kernel: LustreError: 30301:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d26c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 13:26:03 oak-gw06 kernel: LustreError: 30301:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 13:26:03 oak-gw06 kernel: LustreError: 30301:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d26c0) refcount = 2 Jul 25 13:26:03 oak-gw06 kernel: LustreError: 30301:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 13:26:03 oak-gw06 kernel: LustreError: 30301:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ee00/0xf077f1a829b58532 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aabb8bf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 13:26:03 oak-gw06 kernel: LustreError: 30301:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 13:31:11 oak-gw06 kernel: LustreError: 30312:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2d80) refcount = 2 Jul 25 13:31:11 oak-gw06 kernel: LustreError: 30312:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 13:31:11 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 13:31:11 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 13:36:20 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 13:36:20 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 13:36:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501014680, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70a00/0xf077f1a829b5856a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aacab1f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 13:36:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 13:36:20 oak-gw06 kernel: LustreError: 30315:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2feab40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 13:36:20 oak-gw06 kernel: LustreError: 30315:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 13:36:20 oak-gw06 kernel: LustreError: 30315:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2feab40) refcount = 2 Jul 25 13:36:20 oak-gw06 kernel: LustreError: 30315:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 13:36:20 oak-gw06 kernel: LustreError: 30315:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70a00/0xf077f1a829b5856a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aacab1f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 13:36:20 oak-gw06 kernel: LustreError: 30315:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 13:41:28 oak-gw06 kernel: LustreError: 30325:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea0c0) refcount = 2 Jul 25 13:41:28 oak-gw06 kernel: LustreError: 30325:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 13:41:28 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 13:41:28 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 13:46:34 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 13:46:34 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 13:46:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501015294, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73400/0xf077f1a829b585a9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aad9d39 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 13:46:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 13:46:34 oak-gw06 kernel: LustreError: 30328:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2feae40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 13:46:34 oak-gw06 kernel: LustreError: 30328:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 13:46:34 oak-gw06 kernel: LustreError: 30328:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2feae40) refcount = 2 Jul 25 13:46:34 oak-gw06 kernel: LustreError: 30328:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 13:46:34 oak-gw06 kernel: LustreError: 30328:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73400/0xf077f1a829b585a9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aad9d39 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 13:46:34 oak-gw06 kernel: LustreError: 30328:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 13:51:42 oak-gw06 kernel: LustreError: 30339:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea0c0) refcount = 2 Jul 25 13:51:42 oak-gw06 kernel: LustreError: 30339:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 13:51:42 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 13:51:42 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 13:56:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 13:56:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 13:56:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501015909, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73000/0xf077f1a829b585e1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aae8edc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 13:56:49 oak-gw06 kernel: LustreError: 30342:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2feacc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 13:56:49 oak-gw06 kernel: LustreError: 30342:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 13:56:49 oak-gw06 kernel: LustreError: 30342:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2feacc0) refcount = 2 Jul 25 13:56:49 oak-gw06 kernel: LustreError: 30342:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 13:56:49 oak-gw06 kernel: LustreError: 30342:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73000/0xf077f1a829b585e1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aae8edc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 13:56:49 oak-gw06 kernel: LustreError: 30342:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 13:56:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 14:01:55 oak-gw06 kernel: LustreError: 30386:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea3c0) refcount = 2 Jul 25 14:01:55 oak-gw06 kernel: LustreError: 30386:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 14:01:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 14:01:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 14:07:01 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 14:07:01 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 14:07:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501016521, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71000/0xf077f1a829b58620 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aaf7dc3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 14:07:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 14:07:01 oak-gw06 kernel: LustreError: 30389:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880126994e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 14:07:01 oak-gw06 kernel: LustreError: 30389:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 14:07:01 oak-gw06 kernel: LustreError: 30389:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994e40) refcount = 2 Jul 25 14:07:01 oak-gw06 kernel: LustreError: 30389:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 14:07:01 oak-gw06 kernel: LustreError: 30389:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71000/0xf077f1a829b58620 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aaf7dc3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 14:07:01 oak-gw06 kernel: LustreError: 30389:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 14:12:07 oak-gw06 kernel: LustreError: 30399:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994300) refcount = 2 Jul 25 14:12:07 oak-gw06 kernel: LustreError: 30399:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 14:12:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 14:12:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 14:17:17 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 14:17:17 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 14:17:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501017137, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71c00/0xf077f1a829b58658 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ab06f19 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 14:17:17 oak-gw06 kernel: LustreError: 30402:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880126994900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 14:17:17 oak-gw06 kernel: LustreError: 30402:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 14:17:17 oak-gw06 kernel: LustreError: 30402:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994900) refcount = 2 Jul 25 14:17:17 oak-gw06 kernel: LustreError: 30402:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 14:17:17 oak-gw06 kernel: LustreError: 30402:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71c00/0xf077f1a829b58658 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ab06f19 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 14:17:17 oak-gw06 kernel: LustreError: 30402:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 14:17:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 14:22:23 oak-gw06 kernel: LustreError: 30413:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994f00) refcount = 2 Jul 25 14:22:23 oak-gw06 kernel: LustreError: 30413:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 14:22:23 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 14:22:23 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 14:27:30 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 14:27:30 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 14:27:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501017750, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73400/0xf077f1a829b58697 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ab15f18 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 14:27:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 14:27:30 oak-gw06 kernel: LustreError: 30416:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801269949c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 14:27:30 oak-gw06 kernel: LustreError: 30416:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 14:27:30 oak-gw06 kernel: LustreError: 30416:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801269949c0) refcount = 2 Jul 25 14:27:30 oak-gw06 kernel: LustreError: 30416:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 14:27:30 oak-gw06 kernel: LustreError: 30416:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73400/0xf077f1a829b58697 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ab15f18 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 14:27:30 oak-gw06 kernel: LustreError: 30416:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 14:32:36 oak-gw06 kernel: LustreError: 30426:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994d80) refcount = 2 Jul 25 14:32:36 oak-gw06 kernel: LustreError: 30426:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 14:32:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 14:32:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 14:37:46 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 14:37:46 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 14:37:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501018366, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70a00/0xf077f1a829b586d6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ab24ff0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 14:37:46 oak-gw06 kernel: LustreError: 30429:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880126994c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 14:37:46 oak-gw06 kernel: LustreError: 30429:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 14:37:46 oak-gw06 kernel: LustreError: 30429:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994c00) refcount = 2 Jul 25 14:37:46 oak-gw06 kernel: LustreError: 30429:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 14:37:46 oak-gw06 kernel: LustreError: 30429:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70a00/0xf077f1a829b586d6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ab24ff0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 14:37:46 oak-gw06 kernel: LustreError: 30429:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 14:37:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 14:42:54 oak-gw06 kernel: LustreError: 30439:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994000) refcount = 2 Jul 25 14:42:54 oak-gw06 kernel: LustreError: 30439:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 14:42:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 14:42:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 14:48:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 14:48:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 14:48:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501018982, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72600/0xf077f1a829b5870e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ab34322 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 14:48:02 oak-gw06 kernel: LustreError: 30442:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880126994780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 14:48:02 oak-gw06 kernel: LustreError: 30442:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 14:48:02 oak-gw06 kernel: LustreError: 30442:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994780) refcount = 2 Jul 25 14:48:02 oak-gw06 kernel: LustreError: 30442:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 14:48:02 oak-gw06 kernel: LustreError: 30442:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72600/0xf077f1a829b5870e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ab34322 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 14:48:02 oak-gw06 kernel: LustreError: 30442:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 14:48:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 14:53:11 oak-gw06 kernel: LustreError: 30452:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801269949c0) refcount = 2 Jul 25 14:53:11 oak-gw06 kernel: LustreError: 30452:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 14:53:11 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 14:53:11 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 14:58:17 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 14:58:17 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 14:58:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501019597, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72200/0xf077f1a829b58746 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ab4354a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 14:58:17 oak-gw06 kernel: LustreError: 30455:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880126994300) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 14:58:17 oak-gw06 kernel: LustreError: 30455:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 14:58:17 oak-gw06 kernel: LustreError: 30455:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994300) refcount = 2 Jul 25 14:58:17 oak-gw06 kernel: LustreError: 30455:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 14:58:17 oak-gw06 kernel: LustreError: 30455:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72200/0xf077f1a829b58746 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ab4354a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 14:58:17 oak-gw06 kernel: LustreError: 30455:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 14:58:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 15:03:23 oak-gw06 kernel: LustreError: 30498:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994900) refcount = 2 Jul 25 15:03:23 oak-gw06 kernel: LustreError: 30498:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 15:03:23 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 15:03:23 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 15:08:29 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 15:08:29 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 15:08:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501020209, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72800/0xf077f1a829b58785 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ab52382 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 15:08:29 oak-gw06 kernel: LustreError: 30503:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880126994000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 15:08:29 oak-gw06 kernel: LustreError: 30503:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 15:08:29 oak-gw06 kernel: LustreError: 30503:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994000) refcount = 2 Jul 25 15:08:29 oak-gw06 kernel: LustreError: 30503:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 15:08:29 oak-gw06 kernel: LustreError: 30503:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72800/0xf077f1a829b58785 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ab52382 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 15:08:29 oak-gw06 kernel: LustreError: 30503:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 15:08:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 15:13:36 oak-gw06 kernel: LustreError: 30514:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994540) refcount = 2 Jul 25 15:13:36 oak-gw06 kernel: LustreError: 30514:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 15:13:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 15:13:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 15:18:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 15:18:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 15:18:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501020824, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70800/0xf077f1a829b587c4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ab614a7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 15:18:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 15:18:44 oak-gw06 kernel: LustreError: 30517:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801269940c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 15:18:44 oak-gw06 kernel: LustreError: 30517:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 15:18:44 oak-gw06 kernel: LustreError: 30517:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801269940c0) refcount = 2 Jul 25 15:18:44 oak-gw06 kernel: LustreError: 30517:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 15:18:44 oak-gw06 kernel: LustreError: 30517:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70800/0xf077f1a829b587c4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ab614a7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 15:18:44 oak-gw06 kernel: LustreError: 30517:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 15:23:53 oak-gw06 kernel: LustreError: 30527:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994b40) refcount = 2 Jul 25 15:23:53 oak-gw06 kernel: LustreError: 30527:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 15:23:53 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 15:23:53 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 15:28:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 15:28:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 15:28:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501021439, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1800/0xf077f1a829b587fc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ab7057f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 15:28:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 15:28:59 oak-gw06 kernel: LustreError: 30530:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88033d37c180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 15:28:59 oak-gw06 kernel: LustreError: 30530:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 15:28:59 oak-gw06 kernel: LustreError: 30530:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37c180) refcount = 2 Jul 25 15:28:59 oak-gw06 kernel: LustreError: 30530:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 15:28:59 oak-gw06 kernel: LustreError: 30530:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1800/0xf077f1a829b587fc lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ab7057f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 15:28:59 oak-gw06 kernel: LustreError: 30530:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 15:34:04 oak-gw06 kernel: LustreError: 30541:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37cf00) refcount = 2 Jul 25 15:34:04 oak-gw06 kernel: LustreError: 30541:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 15:34:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 15:34:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 15:39:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 15:39:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 15:39:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501022054, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0000/0xf077f1a829b58842 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ab7f722 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 15:39:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 15:39:14 oak-gw06 kernel: LustreError: 30544:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ec3849c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 15:39:14 oak-gw06 kernel: LustreError: 30544:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 15:39:14 oak-gw06 kernel: LustreError: 30544:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec3849c0) refcount = 2 Jul 25 15:39:14 oak-gw06 kernel: LustreError: 30544:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 15:39:14 oak-gw06 kernel: LustreError: 30544:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0000/0xf077f1a829b58842 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ab7f722 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 15:39:14 oak-gw06 kernel: LustreError: 30544:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 15:44:20 oak-gw06 kernel: LustreError: 30555:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384c00) refcount = 2 Jul 25 15:44:20 oak-gw06 kernel: LustreError: 30555:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 15:44:20 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 15:44:20 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 15:49:26 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 15:49:26 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 15:49:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501022666, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0c00/0xf077f1a829b5887a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ab8e736 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 15:49:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 15:49:26 oak-gw06 kernel: LustreError: 30558:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ec384c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 15:49:26 oak-gw06 kernel: LustreError: 30558:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 15:49:26 oak-gw06 kernel: LustreError: 30558:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384c00) refcount = 2 Jul 25 15:49:26 oak-gw06 kernel: LustreError: 30558:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 15:49:26 oak-gw06 kernel: LustreError: 30558:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0c00/0xf077f1a829b5887a lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ab8e736 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 15:49:26 oak-gw06 kernel: LustreError: 30558:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 15:54:36 oak-gw06 kernel: LustreError: 30568:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384cc0) refcount = 2 Jul 25 15:54:36 oak-gw06 kernel: LustreError: 30568:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 15:54:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 15:54:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 15:59:43 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 15:59:43 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 15:59:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501023283, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0c00/0xf077f1a829b588b9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ab9d9f1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 15:59:43 oak-gw06 kernel: LustreError: 30571:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88033d37c600) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 15:59:43 oak-gw06 kernel: LustreError: 30571:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 15:59:43 oak-gw06 kernel: LustreError: 30571:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37c600) refcount = 2 Jul 25 15:59:43 oak-gw06 kernel: LustreError: 30571:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 15:59:43 oak-gw06 kernel: LustreError: 30571:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0c00/0xf077f1a829b588b9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ab9d9f1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 15:59:43 oak-gw06 kernel: LustreError: 30571:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 15:59:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 16:04:51 oak-gw06 kernel: LustreError: 30615:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37c480) refcount = 2 Jul 25 16:04:51 oak-gw06 kernel: LustreError: 30615:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 16:04:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 16:04:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 16:09:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 16:09:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 16:09:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501023897, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0e00/0xf077f1a829b588f8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6abacb2b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 16:09:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 16:09:57 oak-gw06 kernel: LustreError: 30618:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ec384c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 16:09:57 oak-gw06 kernel: LustreError: 30618:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 16:09:57 oak-gw06 kernel: LustreError: 30618:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384c00) refcount = 2 Jul 25 16:09:57 oak-gw06 kernel: LustreError: 30618:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 16:09:57 oak-gw06 kernel: LustreError: 30618:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0e00/0xf077f1a829b588f8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6abacb2b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 16:09:57 oak-gw06 kernel: LustreError: 30618:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 16:15:04 oak-gw06 kernel: LustreError: 30628:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37c9c0) refcount = 2 Jul 25 16:15:04 oak-gw06 kernel: LustreError: 30628:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 16:15:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 16:15:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 16:20:12 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 16:20:12 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 16:20:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501024512, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2600/0xf077f1a829b58930 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6abbbf21 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 16:20:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 16:20:12 oak-gw06 kernel: LustreError: 30638:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88033d37c6c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 16:20:12 oak-gw06 kernel: LustreError: 30638:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 16:20:12 oak-gw06 kernel: LustreError: 30638:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37c6c0) refcount = 2 Jul 25 16:20:12 oak-gw06 kernel: LustreError: 30638:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 16:20:12 oak-gw06 kernel: LustreError: 30638:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2600/0xf077f1a829b58930 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6abbbf21 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 16:20:12 oak-gw06 kernel: LustreError: 30638:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 16:25:21 oak-gw06 kernel: LustreError: 30641:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384e40) refcount = 2 Jul 25 16:25:21 oak-gw06 kernel: LustreError: 30641:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 16:25:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 16:25:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 16:30:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 16:30:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 16:30:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501025128, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0e00/0xf077f1a829b58968 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6abcb1ff expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 16:30:28 oak-gw06 kernel: LustreError: 30652:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ec384480) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 16:30:28 oak-gw06 kernel: LustreError: 30652:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 16:30:28 oak-gw06 kernel: LustreError: 30652:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384480) refcount = 2 Jul 25 16:30:28 oak-gw06 kernel: LustreError: 30652:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 16:30:28 oak-gw06 kernel: LustreError: 30652:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0e00/0xf077f1a829b58968 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6abcb1ff expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 16:30:28 oak-gw06 kernel: LustreError: 30652:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 16:30:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 16:35:35 oak-gw06 kernel: LustreError: 30655:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37cf00) refcount = 2 Jul 25 16:35:35 oak-gw06 kernel: LustreError: 30655:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 16:35:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 16:35:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 16:40:43 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 16:40:43 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 16:40:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501025743, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0c00/0xf077f1a829b589a7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6abda4ba expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 16:40:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 16:40:43 oak-gw06 kernel: LustreError: 30665:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88033d37ce40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 16:40:43 oak-gw06 kernel: LustreError: 30665:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 16:40:43 oak-gw06 kernel: LustreError: 30665:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37ce40) refcount = 2 Jul 25 16:40:43 oak-gw06 kernel: LustreError: 30665:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 16:40:43 oak-gw06 kernel: LustreError: 30665:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0c00/0xf077f1a829b589a7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6abda4ba expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 16:40:43 oak-gw06 kernel: LustreError: 30665:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 16:45:53 oak-gw06 kernel: LustreError: 30679:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec3849c0) refcount = 2 Jul 25 16:45:53 oak-gw06 kernel: LustreError: 30679:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 16:45:53 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 16:45:53 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 16:50:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 16:50:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 16:50:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501026359, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2000/0xf077f1a829b589df lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6abe97f3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 16:50:59 oak-gw06 kernel: LustreError: 30690:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ec384780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 16:50:59 oak-gw06 kernel: LustreError: 30690:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 16:50:59 oak-gw06 kernel: LustreError: 30690:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384780) refcount = 2 Jul 25 16:50:59 oak-gw06 kernel: LustreError: 30690:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 16:50:59 oak-gw06 kernel: LustreError: 30690:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2000/0xf077f1a829b589df lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6abe97f3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 16:50:59 oak-gw06 kernel: LustreError: 30690:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 16:50:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 16:56:06 oak-gw06 kernel: LustreError: 30693:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37ca80) refcount = 2 Jul 25 16:56:06 oak-gw06 kernel: LustreError: 30693:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 16:56:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 16:56:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 17:01:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 17:01:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 17:01:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501026974, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1000/0xf077f1a829b58a17 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6abf8a22 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 17:01:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 17:01:14 oak-gw06 kernel: LustreError: 30736:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88033d37c300) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 17:01:14 oak-gw06 kernel: LustreError: 30736:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 17:01:14 oak-gw06 kernel: LustreError: 30736:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37c300) refcount = 2 Jul 25 17:01:14 oak-gw06 kernel: LustreError: 30736:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 17:01:14 oak-gw06 kernel: LustreError: 30736:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1000/0xf077f1a829b58a17 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6abf8a22 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 17:01:14 oak-gw06 kernel: LustreError: 30736:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 17:06:22 oak-gw06 kernel: LustreError: 30739:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384f00) refcount = 2 Jul 25 17:06:22 oak-gw06 kernel: LustreError: 30739:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 17:06:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 17:06:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 17:11:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 17:11:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 17:11:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501027588, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2a00/0xf077f1a829b58a4f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ac079c6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 17:11:28 oak-gw06 kernel: LustreError: 30749:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ec384600) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 17:11:28 oak-gw06 kernel: LustreError: 30749:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 17:11:28 oak-gw06 kernel: LustreError: 30749:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384600) refcount = 2 Jul 25 17:11:28 oak-gw06 kernel: LustreError: 30749:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 17:11:28 oak-gw06 kernel: LustreError: 30749:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2a00/0xf077f1a829b58a4f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ac079c6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 17:11:28 oak-gw06 kernel: LustreError: 30749:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 17:11:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 17:16:35 oak-gw06 kernel: LustreError: 30752:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37ccc0) refcount = 2 Jul 25 17:16:35 oak-gw06 kernel: LustreError: 30752:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 17:16:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 17:16:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 17:21:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 17:21:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 17:21:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501028201, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2a00/0xf077f1a829b58a87 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ac16a51 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 17:21:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 17:21:41 oak-gw06 kernel: LustreError: 30762:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88033d37c480) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 17:21:41 oak-gw06 kernel: LustreError: 30762:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 17:21:41 oak-gw06 kernel: LustreError: 30762:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37c480) refcount = 2 Jul 25 17:21:41 oak-gw06 kernel: LustreError: 30762:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 17:21:41 oak-gw06 kernel: LustreError: 30762:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2a00/0xf077f1a829b58a87 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ac16a51 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 17:21:41 oak-gw06 kernel: LustreError: 30762:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 17:26:48 oak-gw06 kernel: LustreError: 30765:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384180) refcount = 2 Jul 25 17:26:48 oak-gw06 kernel: LustreError: 30765:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 17:26:48 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 17:26:48 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 17:31:58 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 17:31:58 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 17:31:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501028818, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2600/0xf077f1a829b58abf lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ac25cbf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 17:31:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 17:31:58 oak-gw06 kernel: LustreError: 30775:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ec384240) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 17:31:58 oak-gw06 kernel: LustreError: 30775:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 17:31:58 oak-gw06 kernel: LustreError: 30775:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384240) refcount = 2 Jul 25 17:31:58 oak-gw06 kernel: LustreError: 30775:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 17:31:58 oak-gw06 kernel: LustreError: 30775:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2600/0xf077f1a829b58abf lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ac25cbf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 17:31:58 oak-gw06 kernel: LustreError: 30775:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 17:37:08 oak-gw06 kernel: LustreError: 30778:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37c9c0) refcount = 2 Jul 25 17:37:08 oak-gw06 kernel: LustreError: 30778:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 17:37:08 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 17:37:08 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 17:42:17 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 17:42:17 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 17:42:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501029437, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1a00/0xf077f1a829b58af7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ac352b4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 17:42:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 17:42:17 oak-gw06 kernel: LustreError: 30788:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88033d37c780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 17:42:17 oak-gw06 kernel: LustreError: 30788:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 17:42:17 oak-gw06 kernel: LustreError: 30788:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37c780) refcount = 2 Jul 25 17:42:17 oak-gw06 kernel: LustreError: 30788:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 17:42:17 oak-gw06 kernel: LustreError: 30788:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1a00/0xf077f1a829b58af7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ac352b4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 17:42:17 oak-gw06 kernel: LustreError: 30788:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 17:47:26 oak-gw06 kernel: LustreError: 30791:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec3843c0) refcount = 2 Jul 25 17:47:26 oak-gw06 kernel: LustreError: 30791:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 17:47:26 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 17:47:26 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 17:52:34 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 17:52:34 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 17:52:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501030054, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2e00/0xf077f1a829b58b2f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ac44609 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 17:52:34 oak-gw06 kernel: LustreError: 30801:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ec384240) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 17:52:34 oak-gw06 kernel: LustreError: 30801:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 17:52:34 oak-gw06 kernel: LustreError: 30801:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384240) refcount = 2 Jul 25 17:52:34 oak-gw06 kernel: LustreError: 30801:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 17:52:34 oak-gw06 kernel: LustreError: 30801:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2e00/0xf077f1a829b58b2f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ac44609 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 17:52:34 oak-gw06 kernel: LustreError: 30801:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 17:52:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 17:57:44 oak-gw06 kernel: LustreError: 30804:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37cc00) refcount = 2 Jul 25 17:57:44 oak-gw06 kernel: LustreError: 30804:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 17:57:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 17:57:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 18:02:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 18:02:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 18:02:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501030669, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1c00/0xf077f1a829b58b67 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ac53934 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 18:02:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 18:02:49 oak-gw06 kernel: LustreError: 30847:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88033d37c900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 18:02:49 oak-gw06 kernel: LustreError: 30847:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 18:02:49 oak-gw06 kernel: LustreError: 30847:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37c900) refcount = 2 Jul 25 18:02:49 oak-gw06 kernel: LustreError: 30847:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 18:02:49 oak-gw06 kernel: LustreError: 30847:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1c00/0xf077f1a829b58b67 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ac53934 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 18:02:49 oak-gw06 kernel: LustreError: 30847:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 18:07:56 oak-gw06 kernel: LustreError: 30850:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ec384a80) refcount = 2 Jul 25 18:07:56 oak-gw06 kernel: LustreError: 30850:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 18:07:56 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 18:07:56 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 18:13:07 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 18:13:07 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 18:13:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501031287, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e800/0xf077f1a829b58b9f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ac62c7b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 18:13:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 18:13:07 oak-gw06 kernel: LustreError: 30860:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88006904fb40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 18:13:07 oak-gw06 kernel: LustreError: 30860:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 18:13:07 oak-gw06 kernel: LustreError: 30860:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88006904fb40) refcount = 2 Jul 25 18:13:07 oak-gw06 kernel: LustreError: 30860:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 18:13:07 oak-gw06 kernel: LustreError: 30860:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e800/0xf077f1a829b58b9f lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ac62c7b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 18:13:07 oak-gw06 kernel: LustreError: 30860:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 18:18:13 oak-gw06 kernel: LustreError: 30863:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88006904ff00) refcount = 2 Jul 25 18:18:13 oak-gw06 kernel: LustreError: 30863:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 18:18:13 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 18:18:13 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 18:23:19 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 18:23:19 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 18:23:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501031899, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326d000/0xf077f1a829b58bde lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ac71d37 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 18:23:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 18:23:19 oak-gw06 kernel: LustreError: 30873:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88004f33a3c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 18:23:19 oak-gw06 kernel: LustreError: 30873:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 18:23:19 oak-gw06 kernel: LustreError: 30873:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88004f33a3c0) refcount = 2 Jul 25 18:23:19 oak-gw06 kernel: LustreError: 30873:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 18:23:19 oak-gw06 kernel: LustreError: 30873:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326d000/0xf077f1a829b58bde lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ac71d37 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 18:23:19 oak-gw06 kernel: LustreError: 30873:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 18:28:28 oak-gw06 kernel: LustreError: 30877:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88004f33ad80) refcount = 2 Jul 25 18:28:28 oak-gw06 kernel: LustreError: 30877:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 18:28:28 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 18:28:28 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 18:33:37 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 18:33:37 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 18:33:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501032517, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73c00/0xf077f1a829b58c16 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ac81348 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 18:33:37 oak-gw06 kernel: LustreError: 30887:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2feaa80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 18:33:37 oak-gw06 kernel: LustreError: 30887:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 18:33:37 oak-gw06 kernel: LustreError: 30887:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2feaa80) refcount = 2 Jul 25 18:33:37 oak-gw06 kernel: LustreError: 30887:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 18:33:37 oak-gw06 kernel: LustreError: 30887:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73c00/0xf077f1a829b58c16 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ac81348 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 18:33:37 oak-gw06 kernel: LustreError: 30887:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 18:33:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 18:38:44 oak-gw06 kernel: LustreError: 30890:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2feaf00) refcount = 2 Jul 25 18:38:44 oak-gw06 kernel: LustreError: 30890:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 18:38:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 18:38:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 18:43:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 18:43:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 18:43:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501033129, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71600/0xf077f1a829b58c4e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ac90378 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 18:43:49 oak-gw06 kernel: LustreError: 30901:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2fead80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 18:43:49 oak-gw06 kernel: LustreError: 30901:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 18:43:49 oak-gw06 kernel: LustreError: 30901:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fead80) refcount = 2 Jul 25 18:43:49 oak-gw06 kernel: LustreError: 30901:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 18:43:49 oak-gw06 kernel: LustreError: 30901:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71600/0xf077f1a829b58c4e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ac90378 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 18:43:49 oak-gw06 kernel: LustreError: 30901:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 18:43:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 18:48:59 oak-gw06 kernel: LustreError: 30904:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea900) refcount = 2 Jul 25 18:48:59 oak-gw06 kernel: LustreError: 30904:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 18:48:59 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 18:48:59 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 18:54:06 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 18:54:06 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 18:54:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501033746, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70a00/0xf077f1a829b58c86 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ac9f728 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 18:54:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 18:54:06 oak-gw06 kernel: LustreError: 30914:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2feaa80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 18:54:06 oak-gw06 kernel: LustreError: 30914:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 18:54:06 oak-gw06 kernel: LustreError: 30914:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2feaa80) refcount = 2 Jul 25 18:54:06 oak-gw06 kernel: LustreError: 30914:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 18:54:06 oak-gw06 kernel: LustreError: 30914:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70a00/0xf077f1a829b58c86 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ac9f728 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 18:54:06 oak-gw06 kernel: LustreError: 30914:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 18:59:13 oak-gw06 kernel: LustreError: 30917:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fead80) refcount = 2 Jul 25 18:59:13 oak-gw06 kernel: LustreError: 30917:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 18:59:13 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 18:59:13 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 19:04:21 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 19:04:21 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 19:04:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501034361, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72a00/0xf077f1a829b58cbe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6acae823 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 19:04:21 oak-gw06 kernel: LustreError: 30960:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea480) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 19:04:21 oak-gw06 kernel: LustreError: 30960:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 19:04:21 oak-gw06 kernel: LustreError: 30960:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea480) refcount = 2 Jul 25 19:04:21 oak-gw06 kernel: LustreError: 30960:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 19:04:21 oak-gw06 kernel: LustreError: 30960:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72a00/0xf077f1a829b58cbe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6acae823 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 19:04:21 oak-gw06 kernel: LustreError: 30960:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 19:04:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 19:09:30 oak-gw06 kernel: LustreError: 30963:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea240) refcount = 2 Jul 25 19:09:30 oak-gw06 kernel: LustreError: 30963:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 19:09:30 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 19:09:30 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 19:14:36 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 19:14:36 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 19:14:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501034976, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72200/0xf077f1a829b58cf6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6acbda0c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 19:14:36 oak-gw06 kernel: LustreError: 30973:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 19:14:36 oak-gw06 kernel: LustreError: 30973:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 19:14:36 oak-gw06 kernel: LustreError: 30973:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea900) refcount = 2 Jul 25 19:14:36 oak-gw06 kernel: LustreError: 30973:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 19:14:36 oak-gw06 kernel: LustreError: 30973:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72200/0xf077f1a829b58cf6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6acbda0c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 19:14:36 oak-gw06 kernel: LustreError: 30973:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 19:14:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 19:19:44 oak-gw06 kernel: LustreError: 30976:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2feac00) refcount = 2 Jul 25 19:19:44 oak-gw06 kernel: LustreError: 30976:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 19:19:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 19:19:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 19:24:52 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 19:24:52 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 19:24:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501035592, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73c00/0xf077f1a829b58d35 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6accccf1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 19:24:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 19:24:52 oak-gw06 kernel: LustreError: 30987:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2feaf00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 19:24:52 oak-gw06 kernel: LustreError: 30987:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 19:24:52 oak-gw06 kernel: LustreError: 30987:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2feaf00) refcount = 2 Jul 25 19:24:52 oak-gw06 kernel: LustreError: 30987:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 19:24:52 oak-gw06 kernel: LustreError: 30987:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73c00/0xf077f1a829b58d35 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6accccf1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 19:24:52 oak-gw06 kernel: LustreError: 30987:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 19:30:02 oak-gw06 kernel: LustreError: 30997:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea240) refcount = 2 Jul 25 19:30:02 oak-gw06 kernel: LustreError: 30997:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 19:30:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 19:30:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 19:35:12 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 19:35:12 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 19:35:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501036212, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71800/0xf077f1a829b58d7b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6acdc261 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 19:35:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 19:35:12 oak-gw06 kernel: LustreError: 31000:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 19:35:12 oak-gw06 kernel: LustreError: 31000:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 19:35:12 oak-gw06 kernel: LustreError: 31000:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea000) refcount = 2 Jul 25 19:35:12 oak-gw06 kernel: LustreError: 31000:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 19:35:12 oak-gw06 kernel: LustreError: 31000:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71800/0xf077f1a829b58d7b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6acdc261 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 19:35:12 oak-gw06 kernel: LustreError: 31000:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 19:40:19 oak-gw06 kernel: LustreError: 31010:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea900) refcount = 2 Jul 25 19:40:19 oak-gw06 kernel: LustreError: 31010:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 19:40:19 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 19:40:19 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 19:45:29 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 19:45:29 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 19:45:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501036829, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70000/0xf077f1a829b58dba lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aceb5fc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 19:45:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 19:45:29 oak-gw06 kernel: LustreError: 31013:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 19:45:29 oak-gw06 kernel: LustreError: 31013:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 19:45:29 oak-gw06 kernel: LustreError: 31013:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea000) refcount = 2 Jul 25 19:45:29 oak-gw06 kernel: LustreError: 31013:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 19:45:29 oak-gw06 kernel: LustreError: 31013:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70000/0xf077f1a829b58dba lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aceb5fc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 19:45:29 oak-gw06 kernel: LustreError: 31013:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 19:50:34 oak-gw06 kernel: LustreError: 31023:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2feaa80) refcount = 2 Jul 25 19:50:34 oak-gw06 kernel: LustreError: 31023:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 19:50:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 19:50:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 19:55:43 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 19:55:43 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 19:55:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501037443, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71600/0xf077f1a829b58df2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6acfa7f3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 19:55:43 oak-gw06 kernel: LustreError: 31026:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880126994d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 19:55:43 oak-gw06 kernel: LustreError: 31026:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 19:55:43 oak-gw06 kernel: LustreError: 31026:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994d80) refcount = 2 Jul 25 19:55:43 oak-gw06 kernel: LustreError: 31026:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 19:55:43 oak-gw06 kernel: LustreError: 31026:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71600/0xf077f1a829b58df2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6acfa7f3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 19:55:43 oak-gw06 kernel: LustreError: 31026:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 19:55:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 20:00:53 oak-gw06 kernel: LustreError: 31036:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880126994540) refcount = 2 Jul 25 20:00:53 oak-gw06 kernel: LustreError: 31036:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 20:00:53 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 20:00:53 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 20:05:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 20:05:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 20:05:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501038059, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73200/0xf077f1a829b58e31 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ad09a29 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 20:05:59 oak-gw06 kernel: LustreError: 31073:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801269943c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 20:05:59 oak-gw06 kernel: LustreError: 31073:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 20:05:59 oak-gw06 kernel: LustreError: 31073:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801269943c0) refcount = 2 Jul 25 20:05:59 oak-gw06 kernel: LustreError: 31073:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 20:05:59 oak-gw06 kernel: LustreError: 31073:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a73200/0xf077f1a829b58e31 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ad09a29 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 20:05:59 oak-gw06 kernel: LustreError: 31073:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 20:05:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 20:11:05 oak-gw06 kernel: LustreError: 31083:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea540) refcount = 2 Jul 25 20:11:05 oak-gw06 kernel: LustreError: 31083:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 20:11:05 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 20:11:05 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 20:16:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 20:16:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 20:16:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501038674, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72e00/0xf077f1a829b58e69 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ad18ac2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 20:16:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 20:16:14 oak-gw06 kernel: LustreError: 31086:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 20:16:14 oak-gw06 kernel: LustreError: 31086:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 20:16:14 oak-gw06 kernel: LustreError: 31086:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea180) refcount = 2 Jul 25 20:16:14 oak-gw06 kernel: LustreError: 31086:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 20:16:14 oak-gw06 kernel: LustreError: 31086:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a72e00/0xf077f1a829b58e69 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ad18ac2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 20:16:14 oak-gw06 kernel: LustreError: 31086:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 20:21:22 oak-gw06 kernel: LustreError: 31097:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea240) refcount = 2 Jul 25 20:21:22 oak-gw06 kernel: LustreError: 31097:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 20:21:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 20:21:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 20:26:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 20:26:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 20:26:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501039288, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e000/0xf077f1a829b58ea8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ad27c9d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 20:26:28 oak-gw06 kernel: LustreError: 31100:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 20:26:28 oak-gw06 kernel: LustreError: 31100:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 20:26:28 oak-gw06 kernel: LustreError: 31100:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46000) refcount = 2 Jul 25 20:26:28 oak-gw06 kernel: LustreError: 31100:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 20:26:28 oak-gw06 kernel: LustreError: 31100:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e000/0xf077f1a829b58ea8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ad27c9d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 20:26:28 oak-gw06 kernel: LustreError: 31100:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 20:26:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 20:31:35 oak-gw06 kernel: LustreError: 31110:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46240) refcount = 2 Jul 25 20:31:35 oak-gw06 kernel: LustreError: 31110:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 20:31:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 20:31:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 20:36:43 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 20:36:43 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 20:36:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501039903, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b58ee7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ad36e39 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 20:36:43 oak-gw06 kernel: LustreError: 31113:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 20:36:43 oak-gw06 kernel: LustreError: 31113:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 20:36:43 oak-gw06 kernel: LustreError: 31113:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46180) refcount = 2 Jul 25 20:36:43 oak-gw06 kernel: LustreError: 31113:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 20:36:43 oak-gw06 kernel: LustreError: 31113:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b58ee7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ad36e39 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 20:36:43 oak-gw06 kernel: LustreError: 31113:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 20:36:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 20:41:50 oak-gw06 kernel: LustreError: 31123:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d466c0) refcount = 2 Jul 25 20:41:50 oak-gw06 kernel: LustreError: 31123:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 20:41:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 20:41:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 20:46:58 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 20:46:58 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 20:46:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501040518, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b58f26 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ad45ea1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 20:46:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 20:46:58 oak-gw06 kernel: LustreError: 31126:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 20:46:58 oak-gw06 kernel: LustreError: 31126:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 20:46:58 oak-gw06 kernel: LustreError: 31126:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46a80) refcount = 2 Jul 25 20:46:58 oak-gw06 kernel: LustreError: 31126:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 20:46:58 oak-gw06 kernel: LustreError: 31126:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b58f26 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ad45ea1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 20:46:58 oak-gw06 kernel: LustreError: 31126:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 20:52:04 oak-gw06 kernel: LustreError: 31136:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46000) refcount = 2 Jul 25 20:52:04 oak-gw06 kernel: LustreError: 31136:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 20:52:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 20:52:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 20:57:10 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 20:57:10 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 20:57:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501041130, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56dc00/0xf077f1a829b58f5e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ad54edf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 20:57:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 20:57:10 oak-gw06 kernel: LustreError: 31140:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88004f33a6c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 20:57:10 oak-gw06 kernel: LustreError: 31140:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 20:57:10 oak-gw06 kernel: LustreError: 31140:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88004f33a6c0) refcount = 2 Jul 25 20:57:10 oak-gw06 kernel: LustreError: 31140:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 20:57:10 oak-gw06 kernel: LustreError: 31140:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56dc00/0xf077f1a829b58f5e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ad54edf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 20:57:10 oak-gw06 kernel: LustreError: 31140:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 21:02:17 oak-gw06 kernel: LustreError: 31182:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88004f33a540) refcount = 2 Jul 25 21:02:17 oak-gw06 kernel: LustreError: 31182:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 21:02:17 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 21:02:17 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 21:07:24 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 21:07:24 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 21:07:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501041744, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56da00/0xf077f1a829b58f96 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ad63f71 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 21:07:24 oak-gw06 kernel: LustreError: 31185:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 21:07:24 oak-gw06 kernel: LustreError: 31185:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 21:07:24 oak-gw06 kernel: LustreError: 31185:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46e40) refcount = 2 Jul 25 21:07:24 oak-gw06 kernel: LustreError: 31185:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 21:07:24 oak-gw06 kernel: LustreError: 31185:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56da00/0xf077f1a829b58f96 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ad63f71 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 21:07:24 oak-gw06 kernel: LustreError: 31185:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 21:07:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 21:12:33 oak-gw06 kernel: LustreError: 31195:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46000) refcount = 2 Jul 25 21:12:33 oak-gw06 kernel: LustreError: 31195:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 21:12:33 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 21:12:33 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 21:17:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 21:17:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 21:17:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501042361, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b58fd5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ad73065 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 21:17:41 oak-gw06 kernel: LustreError: 31198:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d466c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 21:17:41 oak-gw06 kernel: LustreError: 31198:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 21:17:41 oak-gw06 kernel: LustreError: 31198:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d466c0) refcount = 2 Jul 25 21:17:41 oak-gw06 kernel: LustreError: 31198:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 21:17:41 oak-gw06 kernel: LustreError: 31198:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b58fd5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ad73065 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 21:17:41 oak-gw06 kernel: LustreError: 31198:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 21:17:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 21:22:48 oak-gw06 kernel: LustreError: 31208:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46900) refcount = 2 Jul 25 21:22:48 oak-gw06 kernel: LustreError: 31208:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 21:22:48 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 21:22:48 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 21:27:54 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 21:27:54 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 21:27:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501042974, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b59014 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ad821ad expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 21:27:54 oak-gw06 kernel: LustreError: 31211:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 21:27:54 oak-gw06 kernel: LustreError: 31211:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 21:27:54 oak-gw06 kernel: LustreError: 31211:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46d80) refcount = 2 Jul 25 21:27:54 oak-gw06 kernel: LustreError: 31211:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 21:27:54 oak-gw06 kernel: LustreError: 31211:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b59014 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ad821ad expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 21:27:54 oak-gw06 kernel: LustreError: 31211:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 21:27:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 21:33:00 oak-gw06 kernel: LustreError: 31222:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d469c0) refcount = 2 Jul 25 21:33:00 oak-gw06 kernel: LustreError: 31222:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 21:33:00 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 21:33:00 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 21:38:06 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 21:38:06 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 21:38:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501043586, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56da00/0xf077f1a829b59053 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ad911eb expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 21:38:06 oak-gw06 kernel: LustreError: 31225:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d463c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 21:38:06 oak-gw06 kernel: LustreError: 31225:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 21:38:06 oak-gw06 kernel: LustreError: 31225:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d463c0) refcount = 2 Jul 25 21:38:06 oak-gw06 kernel: LustreError: 31225:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 21:38:06 oak-gw06 kernel: LustreError: 31225:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56da00/0xf077f1a829b59053 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ad911eb expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 21:38:06 oak-gw06 kernel: LustreError: 31225:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 21:38:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 21:43:11 oak-gw06 kernel: LustreError: 31235:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46540) refcount = 2 Jul 25 21:43:11 oak-gw06 kernel: LustreError: 31235:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 21:43:11 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 21:43:11 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 21:48:21 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 21:48:21 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 21:48:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501044201, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e600/0xf077f1a829b5908b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ada03f0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 21:48:21 oak-gw06 kernel: LustreError: 31239:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 21:48:21 oak-gw06 kernel: LustreError: 31239:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 21:48:21 oak-gw06 kernel: LustreError: 31239:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46780) refcount = 2 Jul 25 21:48:21 oak-gw06 kernel: LustreError: 31239:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 21:48:21 oak-gw06 kernel: LustreError: 31239:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e600/0xf077f1a829b5908b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ada03f0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 21:48:21 oak-gw06 kernel: LustreError: 31239:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 21:48:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 21:53:26 oak-gw06 kernel: LustreError: 31249:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d460c0) refcount = 2 Jul 25 21:53:26 oak-gw06 kernel: LustreError: 31249:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 21:53:26 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 21:53:26 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 21:58:35 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 21:58:35 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 21:58:35 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501044815, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326d400/0xf077f1a829b590c3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6adaf6a4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 21:58:35 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 21:58:35 oak-gw06 kernel: LustreError: 31252:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 21:58:35 oak-gw06 kernel: LustreError: 31252:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 21:58:35 oak-gw06 kernel: LustreError: 31252:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2780) refcount = 2 Jul 25 21:58:35 oak-gw06 kernel: LustreError: 31252:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 21:58:35 oak-gw06 kernel: LustreError: 31252:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326d400/0xf077f1a829b590c3 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6adaf6a4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 21:58:35 oak-gw06 kernel: LustreError: 31252:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 22:03:44 oak-gw06 kernel: LustreError: 31294:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2a80) refcount = 2 Jul 25 22:03:44 oak-gw06 kernel: LustreError: 31294:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 22:03:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 22:03:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 22:08:51 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 22:08:51 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 22:08:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501045431, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ea00/0xf077f1a829b59102 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6adbe863 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 22:08:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 22:08:51 oak-gw06 kernel: LustreError: 31297:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d26c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 22:08:51 oak-gw06 kernel: LustreError: 31297:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 22:08:51 oak-gw06 kernel: LustreError: 31297:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d26c0) refcount = 2 Jul 25 22:08:51 oak-gw06 kernel: LustreError: 31297:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 22:08:51 oak-gw06 kernel: LustreError: 31297:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ea00/0xf077f1a829b59102 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6adbe863 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 22:08:51 oak-gw06 kernel: LustreError: 31297:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 22:13:57 oak-gw06 kernel: LustreError: 31307:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2cc0) refcount = 2 Jul 25 22:13:57 oak-gw06 kernel: LustreError: 31307:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 22:13:57 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 22:13:57 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 22:19:03 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 22:19:03 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 22:19:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501046043, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326fc00/0xf077f1a829b59141 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6adcd9e3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 22:19:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 22:19:03 oak-gw06 kernel: LustreError: 31310:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2540) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 22:19:03 oak-gw06 kernel: LustreError: 31310:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 22:19:03 oak-gw06 kernel: LustreError: 31310:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2540) refcount = 2 Jul 25 22:19:03 oak-gw06 kernel: LustreError: 31310:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 22:19:03 oak-gw06 kernel: LustreError: 31310:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326fc00/0xf077f1a829b59141 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6adcd9e3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 22:19:03 oak-gw06 kernel: LustreError: 31310:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 22:24:08 oak-gw06 kernel: LustreError: 31320:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2600) refcount = 2 Jul 25 22:24:08 oak-gw06 kernel: LustreError: 31320:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 22:24:08 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 22:24:08 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 22:29:17 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 22:29:17 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 22:29:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501046657, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e000/0xf077f1a829b59179 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6addca59 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 22:29:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 22:29:17 oak-gw06 kernel: LustreError: 31323:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d20c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 22:29:17 oak-gw06 kernel: LustreError: 31323:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 22:29:17 oak-gw06 kernel: LustreError: 31323:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d20c0) refcount = 2 Jul 25 22:29:17 oak-gw06 kernel: LustreError: 31323:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 22:29:17 oak-gw06 kernel: LustreError: 31323:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e000/0xf077f1a829b59179 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6addca59 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 22:29:17 oak-gw06 kernel: LustreError: 31323:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 22:34:25 oak-gw06 kernel: LustreError: 31333:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2480) refcount = 2 Jul 25 22:34:25 oak-gw06 kernel: LustreError: 31333:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 22:34:25 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 22:34:25 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 22:39:35 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 22:39:35 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 22:39:35 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501047275, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ee00/0xf077f1a829b591b8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6adebdd8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 22:39:35 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 22:39:35 oak-gw06 kernel: LustreError: 31336:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 22:39:35 oak-gw06 kernel: LustreError: 31336:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 22:39:35 oak-gw06 kernel: LustreError: 31336:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2c00) refcount = 2 Jul 25 22:39:35 oak-gw06 kernel: LustreError: 31336:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 22:39:35 oak-gw06 kernel: LustreError: 31336:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ee00/0xf077f1a829b591b8 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6adebdd8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 22:39:35 oak-gw06 kernel: LustreError: 31336:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 22:44:42 oak-gw06 kernel: LustreError: 31346:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2f00) refcount = 2 Jul 25 22:44:42 oak-gw06 kernel: LustreError: 31346:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 22:44:42 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 22:44:42 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 22:49:51 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 22:49:51 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 22:49:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501047891, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ee00/0xf077f1a829b591fe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6adfb0fc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 22:49:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 22:49:51 oak-gw06 kernel: LustreError: 31349:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 22:49:51 oak-gw06 kernel: LustreError: 31349:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 22:49:51 oak-gw06 kernel: LustreError: 31349:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2900) refcount = 2 Jul 25 22:49:51 oak-gw06 kernel: LustreError: 31349:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 22:49:51 oak-gw06 kernel: LustreError: 31349:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326ee00/0xf077f1a829b591fe lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6adfb0fc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 22:49:51 oak-gw06 kernel: LustreError: 31349:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 22:55:02 oak-gw06 kernel: LustreError: 31359:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2a80) refcount = 2 Jul 25 22:55:02 oak-gw06 kernel: LustreError: 31359:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 22:55:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 22:55:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 23:00:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 23:00:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 23:00:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501048511, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e000/0xf077f1a829b59244 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ae0a4e4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 23:00:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 23:00:11 oak-gw06 kernel: LustreError: 31369:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 23:00:11 oak-gw06 kernel: LustreError: 31369:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 23:00:11 oak-gw06 kernel: LustreError: 31369:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2c00) refcount = 2 Jul 25 23:00:11 oak-gw06 kernel: LustreError: 31369:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 23:00:11 oak-gw06 kernel: LustreError: 31369:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e000/0xf077f1a829b59244 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ae0a4e4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 23:00:11 oak-gw06 kernel: LustreError: 31369:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 23:05:17 oak-gw06 kernel: LustreError: 31404:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2d80) refcount = 2 Jul 25 23:05:17 oak-gw06 kernel: LustreError: 31404:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 23:05:17 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 23:05:17 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 23:10:26 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 23:10:26 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 23:10:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501049126, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326fe00/0xf077f1a829b5928a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ae19561 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 23:10:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 23:10:26 oak-gw06 kernel: LustreError: 31415:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2840) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 23:10:26 oak-gw06 kernel: LustreError: 31415:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 23:10:26 oak-gw06 kernel: LustreError: 31415:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2840) refcount = 2 Jul 25 23:10:26 oak-gw06 kernel: LustreError: 31415:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 23:10:26 oak-gw06 kernel: LustreError: 31415:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326fe00/0xf077f1a829b5928a lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ae19561 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 23:10:26 oak-gw06 kernel: LustreError: 31415:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 23:15:35 oak-gw06 kernel: LustreError: 31420:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2240) refcount = 2 Jul 25 23:15:35 oak-gw06 kernel: LustreError: 31420:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 23:15:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 23:15:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 23:20:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 23:20:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 23:20:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501049741, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e400/0xf077f1a829b592d0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ae285c2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 23:20:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 23:20:41 oak-gw06 kernel: LustreError: 31430:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 23:20:41 oak-gw06 kernel: LustreError: 31430:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 23:20:41 oak-gw06 kernel: LustreError: 31430:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d2780) refcount = 2 Jul 25 23:20:41 oak-gw06 kernel: LustreError: 31430:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 23:20:41 oak-gw06 kernel: LustreError: 31430:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88008326e400/0xf077f1a829b592d0 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ae285c2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 23:20:41 oak-gw06 kernel: LustreError: 31430:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 23:25:50 oak-gw06 kernel: LustreError: 31433:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e97d29c0) refcount = 2 Jul 25 23:25:50 oak-gw06 kernel: LustreError: 31433:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 23:25:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 23:25:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 23:30:56 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 23:30:56 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 23:30:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501050356, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1000/0xf077f1a829b5930f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ae37718 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 23:30:56 oak-gw06 kernel: LustreError: 31445:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880344a8bf00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 23:30:56 oak-gw06 kernel: LustreError: 31445:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 23:30:56 oak-gw06 kernel: LustreError: 31445:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880344a8bf00) refcount = 2 Jul 25 23:30:56 oak-gw06 kernel: LustreError: 31445:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 23:30:56 oak-gw06 kernel: LustreError: 31445:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1000/0xf077f1a829b5930f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ae37718 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 23:30:56 oak-gw06 kernel: LustreError: 31445:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 23:30:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 23:36:04 oak-gw06 kernel: LustreError: 31448:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37c9c0) refcount = 2 Jul 25 23:36:04 oak-gw06 kernel: LustreError: 31448:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 23:36:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 23:36:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 23:41:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 23:41:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 23:41:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501050971, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2400/0xf077f1a829b59347 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ae4683d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 23:41:11 oak-gw06 kernel: LustreError: 31458:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88033d37ca80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 23:41:11 oak-gw06 kernel: LustreError: 31458:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 23:41:11 oak-gw06 kernel: LustreError: 31458:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37ca80) refcount = 2 Jul 25 23:41:11 oak-gw06 kernel: LustreError: 31458:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 23:41:11 oak-gw06 kernel: LustreError: 31458:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2400/0xf077f1a829b59347 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ae4683d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 23:41:11 oak-gw06 kernel: LustreError: 31458:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 23:41:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 23:46:18 oak-gw06 kernel: LustreError: 31461:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37c6c0) refcount = 2 Jul 25 23:46:18 oak-gw06 kernel: LustreError: 31461:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 23:46:18 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 23:46:18 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 25 23:51:27 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 25 23:51:27 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 25 23:51:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501051587, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0200/0xf077f1a829b5937f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ae559d9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 23:51:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 25 23:51:27 oak-gw06 kernel: LustreError: 31471:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88033d37c300) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 25 23:51:27 oak-gw06 kernel: LustreError: 31471:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 25 23:51:27 oak-gw06 kernel: LustreError: 31471:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37c300) refcount = 2 Jul 25 23:51:27 oak-gw06 kernel: LustreError: 31471:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 23:51:27 oak-gw06 kernel: LustreError: 31471:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0200/0xf077f1a829b5937f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ae559d9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 25 23:51:27 oak-gw06 kernel: LustreError: 31471:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 25 23:56:35 oak-gw06 kernel: LustreError: 31487:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880344a8be40) refcount = 2 Jul 25 23:56:35 oak-gw06 kernel: LustreError: 31487:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 25 23:56:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 25 23:56:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 00:01:43 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 00:01:43 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 00:01:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501052203, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e400/0xf077f1a829b593b7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ae64b1a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 00:01:43 oak-gw06 kernel: LustreError: 31531:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 00:01:43 oak-gw06 kernel: LustreError: 31531:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 00:01:43 oak-gw06 kernel: LustreError: 31531:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46d80) refcount = 2 Jul 26 00:01:43 oak-gw06 kernel: LustreError: 31531:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 00:01:43 oak-gw06 kernel: LustreError: 31531:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e400/0xf077f1a829b593b7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ae64b1a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 00:01:43 oak-gw06 kernel: LustreError: 31531:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 00:01:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 00:06:52 oak-gw06 kernel: LustreError: 31534:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46b40) refcount = 2 Jul 26 00:06:52 oak-gw06 kernel: LustreError: 31534:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 00:06:52 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 00:06:52 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 00:12:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 00:12:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 00:12:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501052822, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e400/0xf077f1a829b593f6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ae73e29 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 00:12:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 00:12:02 oak-gw06 kernel: LustreError: 31545:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46300) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 00:12:02 oak-gw06 kernel: LustreError: 31545:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 00:12:02 oak-gw06 kernel: LustreError: 31545:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46300) refcount = 2 Jul 26 00:12:02 oak-gw06 kernel: LustreError: 31545:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 00:12:02 oak-gw06 kernel: LustreError: 31545:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e400/0xf077f1a829b593f6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ae73e29 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 00:12:02 oak-gw06 kernel: LustreError: 31545:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 00:17:09 oak-gw06 kernel: LustreError: 31548:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46d80) refcount = 2 Jul 26 00:17:09 oak-gw06 kernel: LustreError: 31548:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 00:17:09 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 00:17:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 00:22:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 00:22:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 00:22:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501053436, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fa00/0xf077f1a829b5942e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ae82e6e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 00:22:16 oak-gw06 kernel: LustreError: 31558:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 00:22:16 oak-gw06 kernel: LustreError: 31558:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 00:22:16 oak-gw06 kernel: LustreError: 31558:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46180) refcount = 2 Jul 26 00:22:16 oak-gw06 kernel: LustreError: 31558:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 00:22:16 oak-gw06 kernel: LustreError: 31558:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fa00/0xf077f1a829b5942e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ae82e6e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 00:22:16 oak-gw06 kernel: LustreError: 31558:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 00:22:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 00:27:23 oak-gw06 kernel: LustreError: 31561:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46540) refcount = 2 Jul 26 00:27:23 oak-gw06 kernel: LustreError: 31561:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 00:27:23 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 00:27:23 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 00:32:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 00:32:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 00:32:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501054051, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e000/0xf077f1a829b59466 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6ae920a4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 00:32:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 00:32:31 oak-gw06 kernel: LustreError: 31572:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 00:32:31 oak-gw06 kernel: LustreError: 31572:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 00:32:31 oak-gw06 kernel: LustreError: 31572:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46180) refcount = 2 Jul 26 00:32:31 oak-gw06 kernel: LustreError: 31572:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 00:32:31 oak-gw06 kernel: LustreError: 31572:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e000/0xf077f1a829b59466 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6ae920a4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 00:32:31 oak-gw06 kernel: LustreError: 31572:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 00:37:39 oak-gw06 kernel: LustreError: 31575:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d460c0) refcount = 2 Jul 26 00:37:39 oak-gw06 kernel: LustreError: 31575:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 00:37:39 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 00:37:39 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 00:42:45 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 00:42:45 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 00:42:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501054665, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56cc00/0xf077f1a829b5949e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aea111a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 00:42:45 oak-gw06 kernel: LustreError: 31585:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 00:42:45 oak-gw06 kernel: LustreError: 31585:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 00:42:45 oak-gw06 kernel: LustreError: 31585:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46000) refcount = 2 Jul 26 00:42:45 oak-gw06 kernel: LustreError: 31585:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 00:42:45 oak-gw06 kernel: LustreError: 31585:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56cc00/0xf077f1a829b5949e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aea111a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 00:42:45 oak-gw06 kernel: LustreError: 31585:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 00:42:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 00:47:51 oak-gw06 kernel: LustreError: 31588:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46cc0) refcount = 2 Jul 26 00:47:51 oak-gw06 kernel: LustreError: 31588:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 00:47:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 00:47:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 00:52:56 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 00:52:56 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 00:52:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501055276, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e800/0xf077f1a829b594d6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aeb0094 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 00:52:56 oak-gw06 kernel: LustreError: 31598:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 00:52:56 oak-gw06 kernel: LustreError: 31598:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 00:52:56 oak-gw06 kernel: LustreError: 31598:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46780) refcount = 2 Jul 26 00:52:56 oak-gw06 kernel: LustreError: 31598:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 00:52:56 oak-gw06 kernel: LustreError: 31598:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e800/0xf077f1a829b594d6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aeb0094 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 00:52:56 oak-gw06 kernel: LustreError: 31598:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 00:52:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 00:58:06 oak-gw06 kernel: LustreError: 31601:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d463c0) refcount = 2 Jul 26 00:58:06 oak-gw06 kernel: LustreError: 31601:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 00:58:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 00:58:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 01:03:15 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 01:03:15 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 01:03:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501055895, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f000/0xf077f1a829b59515 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aebf4c9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 01:03:15 oak-gw06 kernel: LustreError: 31646:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 01:03:15 oak-gw06 kernel: LustreError: 31646:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 01:03:15 oak-gw06 kernel: LustreError: 31646:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46180) refcount = 2 Jul 26 01:03:15 oak-gw06 kernel: LustreError: 31646:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 01:03:15 oak-gw06 kernel: LustreError: 31646:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f000/0xf077f1a829b59515 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aebf4c9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 01:03:15 oak-gw06 kernel: LustreError: 31646:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 01:03:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 01:08:22 oak-gw06 kernel: LustreError: 31650:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46600) refcount = 2 Jul 26 01:08:22 oak-gw06 kernel: LustreError: 31650:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 01:08:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 01:08:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 01:13:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 01:13:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 01:13:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501056508, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d000/0xf077f1a829b5954d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aece626 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 01:13:28 oak-gw06 kernel: LustreError: 31661:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d460c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 01:13:28 oak-gw06 kernel: LustreError: 31661:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 01:13:28 oak-gw06 kernel: LustreError: 31661:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d460c0) refcount = 2 Jul 26 01:13:28 oak-gw06 kernel: LustreError: 31661:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 01:13:28 oak-gw06 kernel: LustreError: 31661:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d000/0xf077f1a829b5954d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aece626 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 01:13:28 oak-gw06 kernel: LustreError: 31661:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 01:13:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 01:18:36 oak-gw06 kernel: LustreError: 31664:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46900) refcount = 2 Jul 26 01:18:36 oak-gw06 kernel: LustreError: 31664:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 01:18:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 01:18:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 01:23:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 01:23:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 01:23:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501057122, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b59585 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aedd943 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 01:23:42 oak-gw06 kernel: LustreError: 31675:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46840) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 01:23:42 oak-gw06 kernel: LustreError: 31675:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 01:23:42 oak-gw06 kernel: LustreError: 31675:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46840) refcount = 2 Jul 26 01:23:42 oak-gw06 kernel: LustreError: 31675:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 01:23:42 oak-gw06 kernel: LustreError: 31675:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b59585 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aedd943 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 01:23:42 oak-gw06 kernel: LustreError: 31675:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 01:23:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 01:28:48 oak-gw06 kernel: LustreError: 31678:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46c00) refcount = 2 Jul 26 01:28:48 oak-gw06 kernel: LustreError: 31678:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 01:28:48 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 01:28:48 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 01:33:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 01:33:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 01:33:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501057737, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56da00/0xf077f1a829b595bd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aeecabc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 01:33:57 oak-gw06 kernel: LustreError: 31688:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46480) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 01:33:57 oak-gw06 kernel: LustreError: 31688:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 01:33:57 oak-gw06 kernel: LustreError: 31688:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46480) refcount = 2 Jul 26 01:33:57 oak-gw06 kernel: LustreError: 31688:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 01:33:57 oak-gw06 kernel: LustreError: 31688:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56da00/0xf077f1a829b595bd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aeecabc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 01:33:57 oak-gw06 kernel: LustreError: 31688:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 01:33:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 01:39:05 oak-gw06 kernel: LustreError: 31691:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46e40) refcount = 2 Jul 26 01:39:05 oak-gw06 kernel: LustreError: 31691:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 01:39:05 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 01:39:05 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 01:44:15 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 01:44:15 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 01:44:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501058355, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ee00/0xf077f1a829b595fc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6aefbed5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 01:44:15 oak-gw06 kernel: LustreError: 31702:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46300) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 01:44:15 oak-gw06 kernel: LustreError: 31702:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 01:44:15 oak-gw06 kernel: LustreError: 31702:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46300) refcount = 2 Jul 26 01:44:15 oak-gw06 kernel: LustreError: 31702:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 01:44:15 oak-gw06 kernel: LustreError: 31702:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ee00/0xf077f1a829b595fc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6aefbed5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 01:44:15 oak-gw06 kernel: LustreError: 31702:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 01:44:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 01:49:25 oak-gw06 kernel: LustreError: 31706:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46f00) refcount = 2 Jul 26 01:49:25 oak-gw06 kernel: LustreError: 31706:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 01:49:25 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 01:49:25 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 01:54:32 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 01:54:32 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 01:54:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501058972, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d200/0xf077f1a829b59634 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6af0b1eb expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 01:54:32 oak-gw06 kernel: LustreError: 31716:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46480) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 01:54:32 oak-gw06 kernel: LustreError: 31716:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 01:54:32 oak-gw06 kernel: LustreError: 31716:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46480) refcount = 2 Jul 26 01:54:32 oak-gw06 kernel: LustreError: 31716:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 01:54:32 oak-gw06 kernel: LustreError: 31716:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d200/0xf077f1a829b59634 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6af0b1eb expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 01:54:32 oak-gw06 kernel: LustreError: 31716:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 01:54:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 01:59:39 oak-gw06 kernel: LustreError: 31719:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46000) refcount = 2 Jul 26 01:59:39 oak-gw06 kernel: LustreError: 31719:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 01:59:39 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 01:59:39 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 02:04:46 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 02:04:46 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 02:04:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501059586, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b5966c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6af1a428 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 02:04:46 oak-gw06 kernel: LustreError: 31764:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d463c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 02:04:46 oak-gw06 kernel: LustreError: 31764:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 02:04:46 oak-gw06 kernel: LustreError: 31764:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d463c0) refcount = 2 Jul 26 02:04:46 oak-gw06 kernel: LustreError: 31764:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 02:04:46 oak-gw06 kernel: LustreError: 31764:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b5966c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6af1a428 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 02:04:46 oak-gw06 kernel: LustreError: 31764:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 02:04:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 02:09:52 oak-gw06 kernel: LustreError: 31767:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46a80) refcount = 2 Jul 26 02:09:52 oak-gw06 kernel: LustreError: 31767:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 02:09:52 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 02:09:52 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 02:14:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 02:14:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 02:14:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501060199, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f000/0xf077f1a829b596a4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6af296a4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 02:14:59 oak-gw06 kernel: LustreError: 31777:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 02:14:59 oak-gw06 kernel: LustreError: 31777:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 02:14:59 oak-gw06 kernel: LustreError: 31777:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46cc0) refcount = 2 Jul 26 02:14:59 oak-gw06 kernel: LustreError: 31777:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 02:14:59 oak-gw06 kernel: LustreError: 31777:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f000/0xf077f1a829b596a4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6af296a4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 02:14:59 oak-gw06 kernel: LustreError: 31777:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 02:14:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 02:20:08 oak-gw06 kernel: LustreError: 31787:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46540) refcount = 2 Jul 26 02:20:08 oak-gw06 kernel: LustreError: 31787:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 02:20:08 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 02:20:08 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 02:25:15 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 02:25:15 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 02:25:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501060815, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ea00/0xf077f1a829b596dc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6af388e1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 02:25:15 oak-gw06 kernel: LustreError: 31790:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 02:25:15 oak-gw06 kernel: LustreError: 31790:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 02:25:15 oak-gw06 kernel: LustreError: 31790:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46e40) refcount = 2 Jul 26 02:25:15 oak-gw06 kernel: LustreError: 31790:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 02:25:15 oak-gw06 kernel: LustreError: 31790:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ea00/0xf077f1a829b596dc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6af388e1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 02:25:15 oak-gw06 kernel: LustreError: 31790:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 02:25:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 02:30:24 oak-gw06 kernel: LustreError: 31802:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46cc0) refcount = 2 Jul 26 02:30:24 oak-gw06 kernel: LustreError: 31802:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 02:30:24 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 02:30:24 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 02:35:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 02:35:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 02:35:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501061431, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e600/0xf077f1a829b59714 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6af47b02 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 02:35:31 oak-gw06 kernel: LustreError: 31805:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d460c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 02:35:31 oak-gw06 kernel: LustreError: 31805:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 02:35:31 oak-gw06 kernel: LustreError: 31805:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d460c0) refcount = 2 Jul 26 02:35:31 oak-gw06 kernel: LustreError: 31805:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 02:35:31 oak-gw06 kernel: LustreError: 31805:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e600/0xf077f1a829b59714 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6af47b02 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 02:35:31 oak-gw06 kernel: LustreError: 31805:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 02:35:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 02:40:38 oak-gw06 kernel: LustreError: 31816:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46900) refcount = 2 Jul 26 02:40:38 oak-gw06 kernel: LustreError: 31816:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 02:40:38 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 02:40:38 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 02:45:48 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 02:45:48 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 02:45:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501062048, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56dc00/0xf077f1a829b5974c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6af56f14 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 02:45:48 oak-gw06 kernel: LustreError: 31819:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 02:45:48 oak-gw06 kernel: LustreError: 31819:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 02:45:48 oak-gw06 kernel: LustreError: 31819:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46180) refcount = 2 Jul 26 02:45:48 oak-gw06 kernel: LustreError: 31819:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 02:45:48 oak-gw06 kernel: LustreError: 31819:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56dc00/0xf077f1a829b5974c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6af56f14 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 02:45:48 oak-gw06 kernel: LustreError: 31819:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 02:45:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 02:50:56 oak-gw06 kernel: LustreError: 31831:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46300) refcount = 2 Jul 26 02:50:56 oak-gw06 kernel: LustreError: 31831:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 02:50:56 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 02:50:56 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 02:56:05 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 02:56:05 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 02:56:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501062665, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e000/0xf077f1a829b59784 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6af65f98 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 02:56:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 02:56:05 oak-gw06 kernel: LustreError: 31834:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 02:56:05 oak-gw06 kernel: LustreError: 31834:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 02:56:05 oak-gw06 kernel: LustreError: 31834:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46f00) refcount = 2 Jul 26 02:56:05 oak-gw06 kernel: LustreError: 31834:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 02:56:05 oak-gw06 kernel: LustreError: 31834:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e000/0xf077f1a829b59784 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6af65f98 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 02:56:05 oak-gw06 kernel: LustreError: 31834:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 03:01:10 oak-gw06 kernel: LustreError: 31875:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d469c0) refcount = 2 Jul 26 03:01:10 oak-gw06 kernel: LustreError: 31875:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 03:01:10 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 03:01:10 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 03:06:20 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 03:06:20 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 03:06:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501063280, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ca00/0xf077f1a829b597bc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6af750fc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 03:06:20 oak-gw06 kernel: LustreError: 31878:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46840) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 03:06:20 oak-gw06 kernel: LustreError: 31878:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 03:06:20 oak-gw06 kernel: LustreError: 31878:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46840) refcount = 2 Jul 26 03:06:20 oak-gw06 kernel: LustreError: 31878:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 03:06:20 oak-gw06 kernel: LustreError: 31878:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ca00/0xf077f1a829b597bc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6af750fc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 03:06:20 oak-gw06 kernel: LustreError: 31878:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 03:06:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 03:11:27 oak-gw06 kernel: LustreError: 31886:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46900) refcount = 2 Jul 26 03:11:27 oak-gw06 kernel: LustreError: 31886:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 03:11:27 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 03:11:27 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 03:16:37 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 03:16:37 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 03:16:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501063897, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b597f4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6af84332 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 03:16:37 oak-gw06 kernel: LustreError: 31890:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 03:16:37 oak-gw06 kernel: LustreError: 31890:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 03:16:37 oak-gw06 kernel: LustreError: 31890:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46000) refcount = 2 Jul 26 03:16:37 oak-gw06 kernel: LustreError: 31890:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 03:16:37 oak-gw06 kernel: LustreError: 31890:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b597f4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6af84332 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 03:16:37 oak-gw06 kernel: LustreError: 31890:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 03:16:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 03:21:43 oak-gw06 kernel: LustreError: 31898:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46180) refcount = 2 Jul 26 03:21:43 oak-gw06 kernel: LustreError: 31898:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 03:21:43 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 03:21:43 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 03:26:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 03:26:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 03:26:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501064509, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b5982c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6af93418 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 03:26:49 oak-gw06 kernel: LustreError: 31901:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 03:26:49 oak-gw06 kernel: LustreError: 31901:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 03:26:49 oak-gw06 kernel: LustreError: 31901:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46e40) refcount = 2 Jul 26 03:26:49 oak-gw06 kernel: LustreError: 31901:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 03:26:49 oak-gw06 kernel: LustreError: 31901:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56c400/0xf077f1a829b5982c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6af93418 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 03:26:49 oak-gw06 kernel: LustreError: 31901:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 03:26:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 03:31:57 oak-gw06 kernel: LustreError: 31909:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46540) refcount = 2 Jul 26 03:31:57 oak-gw06 kernel: LustreError: 31909:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 03:31:57 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 03:31:57 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 03:37:06 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 03:37:06 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 03:37:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501065126, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ce00/0xf077f1a829b59864 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6afa26a2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 03:37:06 oak-gw06 kernel: LustreError: 31932:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46600) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 03:37:06 oak-gw06 kernel: LustreError: 31932:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 03:37:06 oak-gw06 kernel: LustreError: 31932:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46600) refcount = 2 Jul 26 03:37:06 oak-gw06 kernel: LustreError: 31932:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 03:37:06 oak-gw06 kernel: LustreError: 31932:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ce00/0xf077f1a829b59864 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6afa26a2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 03:37:06 oak-gw06 kernel: LustreError: 31932:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 03:37:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 03:42:12 oak-gw06 kernel: LustreError: 31940:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46780) refcount = 2 Jul 26 03:42:12 oak-gw06 kernel: LustreError: 31940:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 03:42:12 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 03:42:12 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 03:47:19 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 03:47:19 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 03:47:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501065739, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ea00/0xf077f1a829b5989c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6afb187d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 03:47:19 oak-gw06 kernel: LustreError: 31943:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46240) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 03:47:19 oak-gw06 kernel: LustreError: 31943:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 03:47:19 oak-gw06 kernel: LustreError: 31943:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46240) refcount = 2 Jul 26 03:47:19 oak-gw06 kernel: LustreError: 31943:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 03:47:19 oak-gw06 kernel: LustreError: 31943:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ea00/0xf077f1a829b5989c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6afb187d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 03:47:19 oak-gw06 kernel: LustreError: 31943:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 03:47:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 03:52:29 oak-gw06 kernel: LustreError: 31951:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46300) refcount = 2 Jul 26 03:52:29 oak-gw06 kernel: LustreError: 31951:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 03:52:29 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 03:52:29 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 03:57:37 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 03:57:37 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 03:57:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501066357, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fa00/0xf077f1a829b598d4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6afc0b23 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 03:57:37 oak-gw06 kernel: LustreError: 31960:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d460c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 03:57:37 oak-gw06 kernel: LustreError: 31960:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 03:57:37 oak-gw06 kernel: LustreError: 31960:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d460c0) refcount = 2 Jul 26 03:57:37 oak-gw06 kernel: LustreError: 31960:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 03:57:37 oak-gw06 kernel: LustreError: 31960:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fa00/0xf077f1a829b598d4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6afc0b23 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 03:57:37 oak-gw06 kernel: LustreError: 31960:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 03:57:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 04:02:46 oak-gw06 kernel: LustreError: 32003:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46e40) refcount = 2 Jul 26 04:02:46 oak-gw06 kernel: LustreError: 32003:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 04:02:46 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 04:02:46 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 04:07:54 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 04:07:54 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 04:07:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501066974, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fa00/0xf077f1a829b5990c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6afcfd8a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 04:07:54 oak-gw06 kernel: LustreError: 32006:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d460c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 04:07:54 oak-gw06 kernel: LustreError: 32006:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 04:07:54 oak-gw06 kernel: LustreError: 32006:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d460c0) refcount = 2 Jul 26 04:07:54 oak-gw06 kernel: LustreError: 32006:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 04:07:54 oak-gw06 kernel: LustreError: 32006:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fa00/0xf077f1a829b5990c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6afcfd8a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 04:07:54 oak-gw06 kernel: LustreError: 32006:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 04:07:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 04:13:00 oak-gw06 kernel: LustreError: 32016:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46e40) refcount = 2 Jul 26 04:13:00 oak-gw06 kernel: LustreError: 32016:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 04:13:00 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 04:13:00 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 04:18:07 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 04:18:07 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 04:18:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501067587, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d200/0xf077f1a829b59944 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6afdedb3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 04:18:07 oak-gw06 kernel: LustreError: 32019:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d463c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 04:18:07 oak-gw06 kernel: LustreError: 32019:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 04:18:07 oak-gw06 kernel: LustreError: 32019:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d463c0) refcount = 2 Jul 26 04:18:07 oak-gw06 kernel: LustreError: 32019:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 04:18:07 oak-gw06 kernel: LustreError: 32019:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56d200/0xf077f1a829b59944 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6afdedb3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 04:18:07 oak-gw06 kernel: LustreError: 32019:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 04:18:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 04:23:14 oak-gw06 kernel: LustreError: 32029:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46a80) refcount = 2 Jul 26 04:23:14 oak-gw06 kernel: LustreError: 32029:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 04:23:14 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 04:23:14 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 04:28:19 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 04:28:19 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 04:28:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501068199, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ca00/0xf077f1a829b5997c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6afedd34 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 04:28:19 oak-gw06 kernel: LustreError: 32033:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 04:28:19 oak-gw06 kernel: LustreError: 32033:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 04:28:19 oak-gw06 kernel: LustreError: 32033:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46e40) refcount = 2 Jul 26 04:28:19 oak-gw06 kernel: LustreError: 32033:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 04:28:19 oak-gw06 kernel: LustreError: 32033:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ca00/0xf077f1a829b5997c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6afedd34 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 04:28:19 oak-gw06 kernel: LustreError: 32033:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 04:28:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 04:33:29 oak-gw06 kernel: LustreError: 32044:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46d80) refcount = 2 Jul 26 04:33:29 oak-gw06 kernel: LustreError: 32044:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 04:33:29 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 04:33:29 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 04:38:34 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 04:38:34 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 04:38:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501068814, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fa00/0xf077f1a829b599b4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6affcfa9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 04:38:34 oak-gw06 kernel: LustreError: 32047:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46540) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 04:38:34 oak-gw06 kernel: LustreError: 32047:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 04:38:34 oak-gw06 kernel: LustreError: 32047:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46540) refcount = 2 Jul 26 04:38:34 oak-gw06 kernel: LustreError: 32047:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 04:38:34 oak-gw06 kernel: LustreError: 32047:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56fa00/0xf077f1a829b599b4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6affcfa9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 04:38:34 oak-gw06 kernel: LustreError: 32047:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 04:38:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 04:43:43 oak-gw06 kernel: LustreError: 32058:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46600) refcount = 2 Jul 26 04:43:43 oak-gw06 kernel: LustreError: 32058:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 04:43:43 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 04:43:43 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 04:48:51 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 04:48:51 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 04:48:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501069431, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70400/0xf077f1a829b599ec lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b00c30c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 04:48:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 04:48:51 oak-gw06 kernel: LustreError: 32061:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880344a8b480) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 04:48:51 oak-gw06 kernel: LustreError: 32061:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 04:48:51 oak-gw06 kernel: LustreError: 32061:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880344a8b480) refcount = 2 Jul 26 04:48:51 oak-gw06 kernel: LustreError: 32061:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 04:48:51 oak-gw06 kernel: LustreError: 32061:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70400/0xf077f1a829b599ec lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b00c30c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 04:48:51 oak-gw06 kernel: LustreError: 32061:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 04:54:00 oak-gw06 kernel: LustreError: 32071:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880344a8b780) refcount = 2 Jul 26 04:54:00 oak-gw06 kernel: LustreError: 32071:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 04:54:00 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 04:54:00 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 04:59:10 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 04:59:10 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 04:59:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501070050, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71400/0xf077f1a829b59a2b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b01b614 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 04:59:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 04:59:10 oak-gw06 kernel: LustreError: 32074:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880344a8bc00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 04:59:10 oak-gw06 kernel: LustreError: 32074:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 04:59:10 oak-gw06 kernel: LustreError: 32074:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880344a8bc00) refcount = 2 Jul 26 04:59:10 oak-gw06 kernel: LustreError: 32074:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 04:59:10 oak-gw06 kernel: LustreError: 32074:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71400/0xf077f1a829b59a2b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b01b614 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 04:59:10 oak-gw06 kernel: LustreError: 32074:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 05:04:16 oak-gw06 kernel: LustreError: 32116:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880344a8b240) refcount = 2 Jul 26 05:04:16 oak-gw06 kernel: LustreError: 32116:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 05:04:16 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 05:04:16 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 05:09:22 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 05:09:22 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 05:09:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501070662, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71000/0xf077f1a829b59a63 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b02a778 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 05:09:22 oak-gw06 kernel: LustreError: 32120:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea0c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 05:09:22 oak-gw06 kernel: LustreError: 32120:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 05:09:22 oak-gw06 kernel: LustreError: 32120:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea0c0) refcount = 2 Jul 26 05:09:22 oak-gw06 kernel: LustreError: 32120:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 05:09:22 oak-gw06 kernel: LustreError: 32120:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a71000/0xf077f1a829b59a63 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b02a778 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 05:09:22 oak-gw06 kernel: LustreError: 32120:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 05:09:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 05:14:27 oak-gw06 kernel: LustreError: 32130:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea900) refcount = 2 Jul 26 05:14:27 oak-gw06 kernel: LustreError: 32130:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 05:14:27 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 05:14:27 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 05:19:33 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 05:19:33 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 05:19:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501071273, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70600/0xf077f1a829b59a9b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b039945 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 05:19:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 05:19:33 oak-gw06 kernel: LustreError: 32133:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 05:19:33 oak-gw06 kernel: LustreError: 32133:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 05:19:33 oak-gw06 kernel: LustreError: 32133:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea180) refcount = 2 Jul 26 05:19:33 oak-gw06 kernel: LustreError: 32133:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 05:19:33 oak-gw06 kernel: LustreError: 32133:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70600/0xf077f1a829b59a9b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b039945 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 05:19:33 oak-gw06 kernel: LustreError: 32133:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 05:24:41 oak-gw06 kernel: LustreError: 32143:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2feacc0) refcount = 2 Jul 26 05:24:41 oak-gw06 kernel: LustreError: 32143:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 05:24:41 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 05:24:41 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 05:29:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 05:29:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 05:29:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501071889, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70400/0xf077f1a829b59ad3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b048d11 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 05:29:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 05:29:49 oak-gw06 kernel: LustreError: 32146:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2feae40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 05:29:49 oak-gw06 kernel: LustreError: 32146:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 05:29:49 oak-gw06 kernel: LustreError: 32146:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2feae40) refcount = 2 Jul 26 05:29:49 oak-gw06 kernel: LustreError: 32146:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 05:29:49 oak-gw06 kernel: LustreError: 32146:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70400/0xf077f1a829b59ad3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b048d11 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 05:29:49 oak-gw06 kernel: LustreError: 32146:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 05:34:55 oak-gw06 kernel: LustreError: 32158:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea240) refcount = 2 Jul 26 05:34:55 oak-gw06 kernel: LustreError: 32158:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 05:34:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 05:34:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 05:40:03 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 05:40:03 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 05:40:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501072503, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70e00/0xf077f1a829b59b0b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b057fcc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 05:40:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 05:40:03 oak-gw06 kernel: LustreError: 32168:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 05:40:03 oak-gw06 kernel: LustreError: 32168:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 05:40:03 oak-gw06 kernel: LustreError: 32168:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea900) refcount = 2 Jul 26 05:40:03 oak-gw06 kernel: LustreError: 32168:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 05:40:03 oak-gw06 kernel: LustreError: 32168:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880413a70e00/0xf077f1a829b59b0b lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b057fcc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 05:40:03 oak-gw06 kernel: LustreError: 32168:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 05:45:13 oak-gw06 kernel: LustreError: 32171:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2fea780) refcount = 2 Jul 26 05:45:13 oak-gw06 kernel: LustreError: 32171:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 05:45:13 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 05:45:13 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 05:50:21 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 05:50:21 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 05:50:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501073121, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2e00/0xf077f1a829b59b51 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b06723a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 05:50:21 oak-gw06 kernel: LustreError: 32181:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a240) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 05:50:21 oak-gw06 kernel: LustreError: 32181:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 05:50:21 oak-gw06 kernel: LustreError: 32181:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a240) refcount = 2 Jul 26 05:50:21 oak-gw06 kernel: LustreError: 32181:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 05:50:21 oak-gw06 kernel: LustreError: 32181:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2e00/0xf077f1a829b59b51 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b06723a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 05:50:21 oak-gw06 kernel: LustreError: 32181:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 05:50:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 05:55:30 oak-gw06 kernel: LustreError: 32184:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880418abc240) refcount = 2 Jul 26 05:55:30 oak-gw06 kernel: LustreError: 32184:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 05:55:30 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 05:55:30 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 06:00:38 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 06:00:38 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 06:00:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501073738, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2e00/0xf077f1a829b59b89 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b0765ff expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 06:00:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 06:00:38 oak-gw06 kernel: LustreError: 32194:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880418abc540) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 06:00:38 oak-gw06 kernel: LustreError: 32194:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 06:00:38 oak-gw06 kernel: LustreError: 32194:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880418abc540) refcount = 2 Jul 26 06:00:38 oak-gw06 kernel: LustreError: 32194:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 06:00:38 oak-gw06 kernel: LustreError: 32194:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2e00/0xf077f1a829b59b89 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b0765ff expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 06:00:38 oak-gw06 kernel: LustreError: 32194:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 06:05:44 oak-gw06 kernel: LustreError: 32230:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880418abc900) refcount = 2 Jul 26 06:05:44 oak-gw06 kernel: LustreError: 32230:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 06:05:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 06:05:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 06:10:54 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 06:10:54 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 06:10:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501074354, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f400/0xf077f1a829b59bc1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b085993 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 06:10:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 06:10:54 oak-gw06 kernel: LustreError: 32241:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 06:10:54 oak-gw06 kernel: LustreError: 32241:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 06:10:54 oak-gw06 kernel: LustreError: 32241:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46a80) refcount = 2 Jul 26 06:10:54 oak-gw06 kernel: LustreError: 32241:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 06:10:54 oak-gw06 kernel: LustreError: 32241:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56f400/0xf077f1a829b59bc1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b085993 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 06:10:54 oak-gw06 kernel: LustreError: 32241:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 06:16:01 oak-gw06 kernel: LustreError: 32244:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46d80) refcount = 2 Jul 26 06:16:01 oak-gw06 kernel: LustreError: 32244:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 06:16:01 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 06:16:01 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 06:21:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 06:21:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 06:21:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501074971, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b59bf9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b094d3c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 06:21:11 oak-gw06 kernel: LustreError: 32254:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d466c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 06:21:11 oak-gw06 kernel: LustreError: 32254:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 06:21:11 oak-gw06 kernel: LustreError: 32254:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d466c0) refcount = 2 Jul 26 06:21:11 oak-gw06 kernel: LustreError: 32254:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 06:21:11 oak-gw06 kernel: LustreError: 32254:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56de00/0xf077f1a829b59bf9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b094d3c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 06:21:11 oak-gw06 kernel: LustreError: 32254:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 06:21:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 06:26:19 oak-gw06 kernel: LustreError: 32257:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46a80) refcount = 2 Jul 26 06:26:19 oak-gw06 kernel: LustreError: 32257:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 06:26:19 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 06:26:19 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 06:31:27 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 06:31:27 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 06:31:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501075587, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ec00/0xf077f1a829b59c38 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b0a3f87 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 06:31:27 oak-gw06 kernel: LustreError: 32267:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 06:31:27 oak-gw06 kernel: LustreError: 32267:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 06:31:27 oak-gw06 kernel: LustreError: 32267:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46000) refcount = 2 Jul 26 06:31:27 oak-gw06 kernel: LustreError: 32267:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 06:31:27 oak-gw06 kernel: LustreError: 32267:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56ec00/0xf077f1a829b59c38 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b0a3f87 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 06:31:27 oak-gw06 kernel: LustreError: 32267:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 06:31:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 06:36:35 oak-gw06 kernel: LustreError: 32270:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46840) refcount = 2 Jul 26 06:36:35 oak-gw06 kernel: LustreError: 32270:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 06:36:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 06:36:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 06:41:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 06:41:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 06:41:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501076201, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e600/0xf077f1a829b59c70 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b0b3177 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 06:41:41 oak-gw06 kernel: LustreError: 32280:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880148d46180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 06:41:41 oak-gw06 kernel: LustreError: 32280:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 06:41:41 oak-gw06 kernel: LustreError: 32280:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46180) refcount = 2 Jul 26 06:41:41 oak-gw06 kernel: LustreError: 32280:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 06:41:41 oak-gw06 kernel: LustreError: 32280:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a56e600/0xf077f1a829b59c70 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b0b3177 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 06:41:41 oak-gw06 kernel: LustreError: 32280:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 06:41:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 06:46:51 oak-gw06 kernel: LustreError: 32283:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880148d46c00) refcount = 2 Jul 26 06:46:51 oak-gw06 kernel: LustreError: 32283:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 06:46:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 06:46:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 06:52:01 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 06:52:01 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 06:52:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501076821, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6ee00/0xf077f1a829b59ca8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b0c24c5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 06:52:01 oak-gw06 kernel: LustreError: 32293:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880320243f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 06:52:01 oak-gw06 kernel: LustreError: 32293:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 06:52:01 oak-gw06 kernel: LustreError: 32293:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880320243f00) refcount = 2 Jul 26 06:52:01 oak-gw06 kernel: LustreError: 32293:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 06:52:01 oak-gw06 kernel: LustreError: 32293:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6ee00/0xf077f1a829b59ca8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b0c24c5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 06:52:01 oak-gw06 kernel: LustreError: 32293:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 06:52:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 06:57:09 oak-gw06 kernel: LustreError: 32296:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880320243240) refcount = 2 Jul 26 06:57:09 oak-gw06 kernel: LustreError: 32296:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 06:57:09 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 06:57:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 07:02:15 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 07:02:15 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 07:02:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501077435, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6c200/0xf077f1a829b59ce0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b0d165a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 07:02:15 oak-gw06 kernel: LustreError: 32338:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880320243c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 07:02:15 oak-gw06 kernel: LustreError: 32338:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 07:02:15 oak-gw06 kernel: LustreError: 32338:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880320243c00) refcount = 2 Jul 26 07:02:15 oak-gw06 kernel: LustreError: 32338:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 07:02:15 oak-gw06 kernel: LustreError: 32338:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6c200/0xf077f1a829b59ce0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b0d165a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 07:02:15 oak-gw06 kernel: LustreError: 32338:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 07:02:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 07:07:23 oak-gw06 kernel: LustreError: 32341:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880320243e40) refcount = 2 Jul 26 07:07:23 oak-gw06 kernel: LustreError: 32341:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 07:07:23 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 07:07:23 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 07:12:33 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 07:12:33 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 07:12:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501078053, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6ea00/0xf077f1a829b59d18 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b0e091c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 07:12:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 07:12:33 oak-gw06 kernel: LustreError: 32351:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880320243b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 07:12:33 oak-gw06 kernel: LustreError: 32351:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 07:12:33 oak-gw06 kernel: LustreError: 32351:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880320243b40) refcount = 2 Jul 26 07:12:33 oak-gw06 kernel: LustreError: 32351:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 07:12:33 oak-gw06 kernel: LustreError: 32351:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6ea00/0xf077f1a829b59d18 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b0e091c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 07:12:33 oak-gw06 kernel: LustreError: 32351:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 07:17:39 oak-gw06 kernel: LustreError: 32354:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880320243cc0) refcount = 2 Jul 26 07:17:39 oak-gw06 kernel: LustreError: 32354:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 07:17:39 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 07:17:39 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 07:22:45 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 07:22:45 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 07:22:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501078665, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6ee00/0xf077f1a829b59d50 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b0efb36 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 07:22:45 oak-gw06 kernel: LustreError: 32365:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880320243840) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 07:22:45 oak-gw06 kernel: LustreError: 32365:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 07:22:45 oak-gw06 kernel: LustreError: 32365:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880320243840) refcount = 2 Jul 26 07:22:45 oak-gw06 kernel: LustreError: 32365:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 07:22:45 oak-gw06 kernel: LustreError: 32365:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6ee00/0xf077f1a829b59d50 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b0efb36 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 07:22:45 oak-gw06 kernel: LustreError: 32365:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 07:22:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 07:27:51 oak-gw06 kernel: LustreError: 32368:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803202436c0) refcount = 2 Jul 26 07:27:51 oak-gw06 kernel: LustreError: 32368:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 07:27:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 07:27:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 07:33:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 07:33:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 07:33:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501079280, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6f600/0xf077f1a829b59d8f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b0fed1f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 07:33:00 oak-gw06 kernel: LustreError: 32378:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880320243c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 07:33:00 oak-gw06 kernel: LustreError: 32378:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 07:33:00 oak-gw06 kernel: LustreError: 32378:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880320243c00) refcount = 2 Jul 26 07:33:00 oak-gw06 kernel: LustreError: 32378:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 07:33:00 oak-gw06 kernel: LustreError: 32378:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6f600/0xf077f1a829b59d8f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b0fed1f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 07:33:00 oak-gw06 kernel: LustreError: 32378:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 07:33:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 07:38:06 oak-gw06 kernel: LustreError: 32381:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803202439c0) refcount = 2 Jul 26 07:38:06 oak-gw06 kernel: LustreError: 32381:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 07:38:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 07:38:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 07:43:15 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 07:43:15 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 07:43:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501079895, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6ec00/0xf077f1a829b59dce lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b10df47 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 07:43:15 oak-gw06 kernel: LustreError: 32391:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803202433c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 07:43:15 oak-gw06 kernel: LustreError: 32391:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 07:43:15 oak-gw06 kernel: LustreError: 32391:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803202433c0) refcount = 2 Jul 26 07:43:15 oak-gw06 kernel: LustreError: 32391:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 07:43:15 oak-gw06 kernel: LustreError: 32391:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b3f6ec00/0xf077f1a829b59dce lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b10df47 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 07:43:15 oak-gw06 kernel: LustreError: 32391:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 07:43:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 07:48:21 oak-gw06 kernel: LustreError: 32395:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801671673c0) refcount = 2 Jul 26 07:48:21 oak-gw06 kernel: LustreError: 32395:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 07:48:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 07:48:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 07:53:26 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 07:53:26 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 07:53:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501080506, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880197ba4000/0xf077f1a829b59e06 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b11d14c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 07:53:26 oak-gw06 kernel: LustreError: 32405:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88005df08000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 07:53:26 oak-gw06 kernel: LustreError: 32405:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 07:53:26 oak-gw06 kernel: LustreError: 32405:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88005df08000) refcount = 2 Jul 26 07:53:26 oak-gw06 kernel: LustreError: 32405:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 07:53:26 oak-gw06 kernel: LustreError: 32405:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880197ba4000/0xf077f1a829b59e06 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b11d14c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 07:53:26 oak-gw06 kernel: LustreError: 32405:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 07:53:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 07:58:34 oak-gw06 kernel: LustreError: 32408:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88005df086c0) refcount = 2 Jul 26 07:58:34 oak-gw06 kernel: LustreError: 32408:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 07:58:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 07:58:34 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 08:03:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 08:03:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 08:03:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501081121, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880197ba6e00/0xf077f1a829b59e3e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b12c4bd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 08:03:41 oak-gw06 kernel: LustreError: 32450:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88005df08a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 08:03:41 oak-gw06 kernel: LustreError: 32450:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 08:03:41 oak-gw06 kernel: LustreError: 32450:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88005df08a80) refcount = 2 Jul 26 08:03:41 oak-gw06 kernel: LustreError: 32450:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 08:03:41 oak-gw06 kernel: LustreError: 32450:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880197ba6e00/0xf077f1a829b59e3e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b12c4bd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 08:03:41 oak-gw06 kernel: LustreError: 32450:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 08:03:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 08:08:51 oak-gw06 kernel: LustreError: 32453:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803bd54e900) refcount = 2 Jul 26 08:08:51 oak-gw06 kernel: LustreError: 32453:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 08:08:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 08:08:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 08:13:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 08:13:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 08:13:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501081739, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880197ba6e00/0xf077f1a829b59e76 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b13b890 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 08:13:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 08:13:59 oak-gw06 kernel: LustreError: 32463:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803bd54e780) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 08:13:59 oak-gw06 kernel: LustreError: 32463:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 08:13:59 oak-gw06 kernel: LustreError: 32463:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803bd54e780) refcount = 2 Jul 26 08:13:59 oak-gw06 kernel: LustreError: 32463:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 08:13:59 oak-gw06 kernel: LustreError: 32463:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880197ba6e00/0xf077f1a829b59e76 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b13b890 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 08:13:59 oak-gw06 kernel: LustreError: 32463:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 08:19:05 oak-gw06 kernel: LustreError: 32466:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803bd54e600) refcount = 2 Jul 26 08:19:05 oak-gw06 kernel: LustreError: 32466:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 08:19:05 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 08:19:05 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 08:24:15 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 08:24:15 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 08:24:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501082355, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1e00/0xf077f1a829b59eae lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b14ac1d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 08:24:15 oak-gw06 kernel: LustreError: 32476:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880418abc900) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 08:24:15 oak-gw06 kernel: LustreError: 32476:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 08:24:15 oak-gw06 kernel: LustreError: 32476:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880418abc900) refcount = 2 Jul 26 08:24:15 oak-gw06 kernel: LustreError: 32476:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 08:24:15 oak-gw06 kernel: LustreError: 32476:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1e00/0xf077f1a829b59eae lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b14ac1d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 08:24:15 oak-gw06 kernel: LustreError: 32476:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 08:24:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 08:29:20 oak-gw06 kernel: LustreError: 32480:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007ea6ac00) refcount = 2 Jul 26 08:29:20 oak-gw06 kernel: LustreError: 32480:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 08:29:20 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 08:29:20 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 08:34:29 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 08:34:29 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 08:34:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501082969, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1a00/0xf077f1a829b59ee6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b159f10 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 08:34:29 oak-gw06 kernel: LustreError: 32491:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a9c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 08:34:29 oak-gw06 kernel: LustreError: 32491:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 08:34:29 oak-gw06 kernel: LustreError: 32491:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a9c0) refcount = 2 Jul 26 08:34:29 oak-gw06 kernel: LustreError: 32491:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 08:34:29 oak-gw06 kernel: LustreError: 32491:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1a00/0xf077f1a829b59ee6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b159f10 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 08:34:29 oak-gw06 kernel: LustreError: 32491:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 08:34:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 08:39:37 oak-gw06 kernel: LustreError: 32494:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007ea6ad80) refcount = 2 Jul 26 08:39:37 oak-gw06 kernel: LustreError: 32494:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 08:39:37 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 08:39:37 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 08:44:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 08:44:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 08:44:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501083582, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0a00/0xf077f1a829b59f1e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b1691d2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 08:44:42 oak-gw06 kernel: LustreError: 32504:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a840) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 08:44:42 oak-gw06 kernel: LustreError: 32504:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 08:44:42 oak-gw06 kernel: LustreError: 32504:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a840) refcount = 2 Jul 26 08:44:42 oak-gw06 kernel: LustreError: 32504:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 08:44:42 oak-gw06 kernel: LustreError: 32504:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0a00/0xf077f1a829b59f1e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b1691d2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 08:44:42 oak-gw06 kernel: LustreError: 32504:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 08:44:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 08:49:51 oak-gw06 kernel: LustreError: 32507:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007ea6af00) refcount = 2 Jul 26 08:49:51 oak-gw06 kernel: LustreError: 32507:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 08:49:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 08:49:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 08:55:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 08:55:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 08:55:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501084200, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0a00/0xf077f1a829b59f56 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b178455 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 08:55:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 08:55:00 oak-gw06 kernel: LustreError: 32517:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a000) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 08:55:00 oak-gw06 kernel: LustreError: 32517:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 08:55:00 oak-gw06 kernel: LustreError: 32517:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a000) refcount = 2 Jul 26 08:55:00 oak-gw06 kernel: LustreError: 32517:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 08:55:00 oak-gw06 kernel: LustreError: 32517:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0a00/0xf077f1a829b59f56 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b178455 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 08:55:00 oak-gw06 kernel: LustreError: 32517:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 09:00:06 oak-gw06 kernel: LustreError: 32528:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880418abc900) refcount = 2 Jul 26 09:00:06 oak-gw06 kernel: LustreError: 32528:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 09:00:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 09:00:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 09:05:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 09:05:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 09:05:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501084813, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2800/0xf077f1a829b59f8e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b1876c3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 09:05:13 oak-gw06 kernel: LustreError: 32563:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a0c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 09:05:13 oak-gw06 kernel: LustreError: 32563:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 09:05:13 oak-gw06 kernel: LustreError: 32563:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a0c0) refcount = 2 Jul 26 09:05:13 oak-gw06 kernel: LustreError: 32563:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 09:05:13 oak-gw06 kernel: LustreError: 32563:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb2800/0xf077f1a829b59f8e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b1876c3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 09:05:13 oak-gw06 kernel: LustreError: 32563:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 09:05:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 09:10:21 oak-gw06 kernel: LustreError: 32574:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a780) refcount = 2 Jul 26 09:10:21 oak-gw06 kernel: LustreError: 32574:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 09:10:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 09:10:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 09:15:30 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 09:15:30 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 09:15:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501085430, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0000/0xf077f1a829b59fcd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b196a57 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 09:15:30 oak-gw06 kernel: LustreError: 32577:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88007ea6af00) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 09:15:30 oak-gw06 kernel: LustreError: 32577:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 09:15:30 oak-gw06 kernel: LustreError: 32577:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007ea6af00) refcount = 2 Jul 26 09:15:30 oak-gw06 kernel: LustreError: 32577:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 09:15:30 oak-gw06 kernel: LustreError: 32577:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0000/0xf077f1a829b59fcd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b196a57 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 09:15:30 oak-gw06 kernel: LustreError: 32577:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 09:15:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 09:20:40 oak-gw06 kernel: LustreError: 32587:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a000) refcount = 2 Jul 26 09:20:40 oak-gw06 kernel: LustreError: 32587:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 09:20:40 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 09:20:40 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Jul 26 09:25:48 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jul 26 09:25:48 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Jul 26 09:25:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1501086048, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1c00/0xf077f1a829b5a005 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x5c44f40f6b1a5d97 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 09:25:48 oak-gw06 kernel: LustreError: 32590:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a300) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 09:25:48 oak-gw06 kernel: LustreError: 32590:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Jul 26 09:25:48 oak-gw06 kernel: LustreError: 32590:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a300) refcount = 2 Jul 26 09:25:48 oak-gw06 kernel: LustreError: 32590:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 09:25:48 oak-gw06 kernel: LustreError: 32590:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb1c00/0xf077f1a829b5a005 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b1a5d97 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 09:25:48 oak-gw06 kernel: LustreError: 32590:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Jul 26 09:25:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Jul 26 09:29:41 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1501086573/real 1501086573] req@ffff8801a16c0600 x1566266001184496/t0(0) o400->MGC10.0.2.51@o2ib5@10.0.2.51@o2ib5:26/25 lens 224/224 e 0 to 1 dl 1501086581 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 26 09:29:41 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 26 09:30:12 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1501086606/real 1501086606] req@ffff88041cb14900 x1566266001185216/t0(0) o250->MGC10.0.2.51@o2ib5@10.0.2.52@o2ib5:26/25 lens 520/544 e 0 to 1 dl 1501086612 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 26 09:30:12 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 26 09:31:07 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1501086656/real 1501086656] req@ffff880212c5c600 x1566266001186624/t0(0) o250->MGC10.0.2.51@o2ib5@10.0.2.52@o2ib5:26/25 lens 520/544 e 0 to 1 dl 1501086667 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 26 09:31:07 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 26 09:32:32 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1501086731/real 1501086731] req@ffff8801243db000 x1566266001188736/t0(0) o250->MGC10.0.2.51@o2ib5@10.0.2.51@o2ib5:26/25 lens 520/544 e 0 to 1 dl 1501086752 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 26 09:32:32 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 26 09:35:07 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1501086881/real 1501086907] req@ffff8804197a4300 x1566266001192928/t0(0) o250->MGC10.0.2.51@o2ib5@10.0.2.51@o2ib5:26/25 lens 520/544 e 0 to 1 dl 1501086912 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jul 26 09:35:07 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 26 09:38:51 oak-gw06 kernel: Lustre: Evicted from MGS (at 10.0.2.51@o2ib5) after server handle changed from 0x5c44f40f6a1b0dbe to 0x7f86053cb7f7e12a Jul 26 09:38:51 oak-gw06 kernel: LustreError: 32601:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a180) refcount nonzero (1) after lock cleanup; forcing cleanup. Jul 26 09:38:51 oak-gw06 kernel: LustreError: 32601:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007ea6a180) refcount = 2 Jul 26 09:38:51 oak-gw06 kernel: LustreError: 32601:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Jul 26 09:38:51 oak-gw06 kernel: LustreError: 32601:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276eb0e00/0xf077f1a829b5a021 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x5c44f40f6b1ad30f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Jul 26 09:38:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Jul 26 09:38:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 1 09:35:12 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 1 09:35:12 oak-gw06 kernel: CPU: 6 PID: 11759 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 09:35:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 09:35:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 09:35:12 oak-gw06 kernel: 00000000000080d0 000000002c9a08aa ffff8801f68a7858 ffffffff8168662f Aug 1 09:35:12 oak-gw06 kernel: ffff8801f68a78e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 1 09:35:12 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff8801f68a78e8 000000002c9a08aa Aug 1 09:35:12 oak-gw06 kernel: Call Trace: Aug 1 09:35:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 09:35:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 09:35:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 09:35:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 09:35:12 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 09:35:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 09:35:12 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 09:35:12 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 09:35:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 09:35:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 09:35:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 09:35:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 09:35:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 09:35:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 09:35:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 09:35:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 09:35:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 09:35:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 09:35:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 09:35:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:35:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 09:35:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:35:12 oak-gw06 kernel: Mem-Info: Aug 1 09:35:12 oak-gw06 kernel: active_anon:40348 inactive_anon:41391 isolated_anon:0#012 active_file:741511 inactive_file:1668438 isolated_file:0#012 unevictable:0 dirty:5176 writeback:6603 unstable:0#012 slab_reclaimable:53754 slab_unreclaimable:945570#012 mapped:4917 shmem:39000 pagetables:1648 bounce:0#012 free:473689 free_pcp:1718 free_cma:0 Aug 1 09:35:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 09:35:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 09:35:12 oak-gw06 kernel: Node 0 DMA32 free:370508kB min:11976kB low:14968kB high:17964kB active_anon:26360kB inactive_anon:31248kB active_file:512404kB inactive_file:1183004kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:6724kB writeback:3988kB mapped:1988kB shmem:31276kB slab_reclaimable:39464kB slab_unreclaimable:660708kB kernel_stack:1024kB pagetables:1068kB unstable:0kB bounce:0kB free_pcp:2844kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:35:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 09:35:12 oak-gw06 kernel: Node 0 Normal free:1502372kB min:55536kB low:69420kB high:83304kB active_anon:135032kB inactive_anon:134316kB active_file:2453640kB inactive_file:5509416kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:26012kB writeback:26016kB mapped:17680kB shmem:124724kB slab_reclaimable:175552kB slab_unreclaimable:3121556kB kernel_stack:4656kB pagetables:5524kB unstable:0kB bounce:0kB free_pcp:3584kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:35:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 09:35:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 09:35:12 oak-gw06 kernel: Node 0 DMA32: 1146*4kB (UEM) 819*8kB (UEM) 2181*16kB (UEM) 4274*32kB (UEM) 2047*64kB (UEM) 393*128kB (UM) 19*256kB (UM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 369488kB Aug 1 09:35:12 oak-gw06 kernel: Node 0 Normal: 5650*4kB (UE) 3708*8kB (UEM) 6273*16kB (UEM) 19382*32kB (UEM) 9378*64kB (UEM) 989*128kB (UEM) 4*256kB (UEM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 1501176kB Aug 1 09:35:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 09:35:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 09:35:12 oak-gw06 kernel: 2111444 total pagecache pages Aug 1 09:35:12 oak-gw06 kernel: 0 pages in swap cache Aug 1 09:35:12 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 09:35:12 oak-gw06 kernel: Free swap = 4194300kB Aug 1 09:35:12 oak-gw06 kernel: Total swap = 4194300kB Aug 1 09:35:12 oak-gw06 kernel: 4194203 pages RAM Aug 1 09:35:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 09:35:12 oak-gw06 kernel: 127313 pages reserved Aug 1 09:35:12 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 1 09:35:12 oak-gw06 kernel: CPU: 6 PID: 11759 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 09:35:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 09:35:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 09:35:12 oak-gw06 kernel: 00000000000080d0 000000002c9a08aa ffff8801f68a7808 ffffffff8168662f Aug 1 09:35:12 oak-gw06 kernel: ffff8801f68a7898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 1 09:35:12 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801f68a7868 000000002c9a08aa Aug 1 09:35:12 oak-gw06 kernel: Call Trace: Aug 1 09:35:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 09:35:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 09:35:12 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 1 09:35:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 09:35:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 09:35:12 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 09:35:12 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 09:35:12 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 09:35:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 09:35:12 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 09:35:12 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 09:35:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 09:35:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 09:35:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 09:35:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 09:35:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 09:35:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 09:35:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 09:35:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 09:35:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 09:35:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 09:35:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 09:35:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:35:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 09:35:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:35:12 oak-gw06 kernel: Mem-Info: Aug 1 09:35:12 oak-gw06 kernel: active_anon:40252 inactive_anon:41487 isolated_anon:0#012 active_file:726955 inactive_file:1681156 isolated_file:0#012 unevictable:0 dirty:8306 writeback:4086 unstable:0#012 slab_reclaimable:53754 slab_unreclaimable:944886#012 mapped:4917 shmem:39000 pagetables:1648 bounce:0#012 free:481866 free_pcp:1169 free_cma:0 Aug 1 09:35:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 09:35:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 09:35:12 oak-gw06 kernel: Node 0 DMA32 free:381360kB min:11976kB low:14968kB high:17964kB active_anon:26360kB inactive_anon:31248kB active_file:496612kB inactive_file:1191664kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:8160kB writeback:1236kB mapped:1988kB shmem:31276kB slab_reclaimable:39464kB slab_unreclaimable:659764kB kernel_stack:1024kB pagetables:1068kB unstable:0kB bounce:0kB free_pcp:2460kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:35:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 09:35:12 oak-gw06 kernel: Node 0 Normal free:1559656kB min:55536kB low:69420kB high:83304kB active_anon:134648kB inactive_anon:134700kB active_file:2373040kB inactive_file:5541656kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:26012kB writeback:12436kB mapped:17680kB shmem:124724kB slab_reclaimable:175552kB slab_unreclaimable:3116660kB kernel_stack:4656kB pagetables:5524kB unstable:0kB bounce:0kB free_pcp:3360kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:35:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 09:35:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 09:35:12 oak-gw06 kernel: Node 0 DMA32: 1179*4kB (UEM) 827*8kB (UEM) 2241*16kB (UEM) 4427*32kB (UEM) 2123*64kB (UEM) 399*128kB (UM) 19*256kB (UM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 381172kB Aug 1 09:35:12 oak-gw06 kernel: Node 0 Normal: 4985*4kB (UE) 3721*8kB (UEM) 6722*16kB (UEM) 20480*32kB (UEM) 9578*64kB (UEM) 1007*128kB (UEM) 4*256kB (UEM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 1556044kB Aug 1 09:35:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 09:35:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 09:35:12 oak-gw06 kernel: 2092490 total pagecache pages Aug 1 09:35:12 oak-gw06 kernel: 0 pages in swap cache Aug 1 09:35:12 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 09:35:12 oak-gw06 kernel: Free swap = 4194300kB Aug 1 09:35:12 oak-gw06 kernel: Total swap = 4194300kB Aug 1 09:35:12 oak-gw06 kernel: 4194203 pages RAM Aug 1 09:35:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 09:35:12 oak-gw06 kernel: 127313 pages reserved Aug 1 09:45:12 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 09:45:12 oak-gw06 kernel: CPU: 3 PID: 11786 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 09:45:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 09:45:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 09:45:12 oak-gw06 kernel: 00000000000080d0 000000004605052a ffff880233cbb858 ffffffff8168662f Aug 1 09:45:12 oak-gw06 kernel: ffff880233cbb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 09:45:12 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880233cbb8b8 000000004605052a Aug 1 09:45:12 oak-gw06 kernel: Call Trace: Aug 1 09:45:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 09:45:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 09:45:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 09:45:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 09:45:12 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 09:45:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 09:45:12 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 09:45:12 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 09:45:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 09:45:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 09:45:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 09:45:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 09:45:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 09:45:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 09:45:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 09:45:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 09:45:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 09:45:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 09:45:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 09:45:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:45:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 09:45:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:45:12 oak-gw06 kernel: Mem-Info: Aug 1 09:45:12 oak-gw06 kernel: active_anon:28460 inactive_anon:41519 isolated_anon:0#012 active_file:1126429 inactive_file:1253907 isolated_file:0#012 unevictable:0 dirty:6478 writeback:7097 unstable:0#012 slab_reclaimable:49518 slab_unreclaimable:944745#012 mapped:4933 shmem:39000 pagetables:1632 bounce:0#012 free:517520 free_pcp:2062 free_cma:0 Aug 1 09:45:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 09:45:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 09:45:12 oak-gw06 kernel: Node 0 DMA32 free:373792kB min:11976kB low:14968kB high:17964kB active_anon:20388kB inactive_anon:31248kB active_file:802096kB inactive_file:894996kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4800kB writeback:2584kB mapped:1992kB shmem:31276kB slab_reclaimable:36256kB slab_unreclaimable:665728kB kernel_stack:1024kB pagetables:1152kB unstable:0kB bounce:0kB free_pcp:3504kB local_pcp:92kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:45:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 09:45:12 oak-gw06 kernel: Node 0 Normal free:1696848kB min:55536kB low:69420kB high:83304kB active_anon:93452kB inactive_anon:134828kB active_file:3679148kB inactive_file:4141204kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:23440kB writeback:13864kB mapped:17740kB shmem:124724kB slab_reclaimable:161816kB slab_unreclaimable:3112420kB kernel_stack:4656kB pagetables:5376kB unstable:0kB bounce:0kB free_pcp:4536kB local_pcp:16kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:45:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 09:45:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 09:45:12 oak-gw06 kernel: Node 0 DMA32: 3384*4kB (UEM) 3567*8kB (UEM) 5687*16kB (UEM) 6744*32kB (UEM) 434*64kB (UEM) 2*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 376904kB Aug 1 09:45:12 oak-gw06 kernel: Node 0 Normal: 12361*4kB (UEM) 20819*8kB (UEM) 30541*16kB (UEM) 26789*32kB (UEM) 2013*64kB (UEM) 37*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1695468kB Aug 1 09:45:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 09:45:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 09:45:12 oak-gw06 kernel: 2084073 total pagecache pages Aug 1 09:45:12 oak-gw06 kernel: 0 pages in swap cache Aug 1 09:45:12 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 09:45:12 oak-gw06 kernel: Free swap = 4194300kB Aug 1 09:45:12 oak-gw06 kernel: Total swap = 4194300kB Aug 1 09:45:12 oak-gw06 kernel: 4194203 pages RAM Aug 1 09:45:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 09:45:12 oak-gw06 kernel: 127313 pages reserved Aug 1 09:45:12 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 09:45:12 oak-gw06 kernel: CPU: 3 PID: 11786 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 09:45:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 09:45:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 09:45:12 oak-gw06 kernel: 00000000000080d0 000000004605052a ffff880233cbb808 ffffffff8168662f Aug 1 09:45:12 oak-gw06 kernel: ffff880233cbb898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 09:45:12 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880233cbb868 000000004605052a Aug 1 09:45:12 oak-gw06 kernel: Call Trace: Aug 1 09:45:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 09:45:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 09:45:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 09:45:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 09:45:12 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 09:45:12 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 09:45:12 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 09:45:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 09:45:12 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 09:45:12 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 09:45:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 09:45:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 09:45:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 09:45:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 09:45:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 09:45:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 09:45:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 09:45:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 09:45:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 09:45:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 09:45:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 09:45:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:45:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 09:45:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:45:12 oak-gw06 kernel: Mem-Info: Aug 1 09:45:12 oak-gw06 kernel: active_anon:28460 inactive_anon:41519 isolated_anon:0#012 active_file:1117353 inactive_file:1281235 isolated_file:0#012 unevictable:0 dirty:13024 writeback:7284 unstable:0#012 slab_reclaimable:49518 slab_unreclaimable:944473#012 mapped:4933 shmem:39000 pagetables:1632 bounce:0#012 free:511181 free_pcp:990 free_cma:0 Aug 1 09:45:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 09:45:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 09:45:12 oak-gw06 kernel: Node 0 DMA32 free:369364kB min:11976kB low:14968kB high:17964kB active_anon:20388kB inactive_anon:31248kB active_file:798584kB inactive_file:912480kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:8532kB writeback:4020kB mapped:1992kB shmem:31276kB slab_reclaimable:36256kB slab_unreclaimable:665736kB kernel_stack:1024kB pagetables:1152kB unstable:0kB bounce:0kB free_pcp:1992kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:45:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 09:45:12 oak-gw06 kernel: Node 0 Normal free:1650768kB min:55536kB low:69420kB high:83304kB active_anon:93452kB inactive_anon:134828kB active_file:3670828kB inactive_file:4220540kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:40384kB writeback:19296kB mapped:17740kB shmem:124724kB slab_reclaimable:161816kB slab_unreclaimable:3112156kB kernel_stack:4656kB pagetables:5376kB unstable:0kB bounce:0kB free_pcp:2676kB local_pcp:40kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:45:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 09:45:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 09:45:12 oak-gw06 kernel: Node 0 DMA32: 1484*4kB (UEM) 2856*8kB (UEM) 5717*16kB (UEM) 6780*32kB (UEM) 437*64kB (UEM) 2*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 365440kB Aug 1 09:45:12 oak-gw06 kernel: Node 0 Normal: 7678*4kB (UE) 13983*8kB (UEM) 30841*16kB (UEM) 26898*32kB (UEM) 2016*64kB (UEM) 37*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1630528kB Aug 1 09:45:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 09:45:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 09:45:12 oak-gw06 kernel: 2107703 total pagecache pages Aug 1 09:45:12 oak-gw06 kernel: 0 pages in swap cache Aug 1 09:45:12 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 09:45:12 oak-gw06 kernel: Free swap = 4194300kB Aug 1 09:45:12 oak-gw06 kernel: Total swap = 4194300kB Aug 1 09:45:12 oak-gw06 kernel: 4194203 pages RAM Aug 1 09:45:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 09:45:12 oak-gw06 kernel: 127313 pages reserved Aug 1 09:50:12 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 09:50:12 oak-gw06 kernel: CPU: 3 PID: 11786 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 09:50:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 09:50:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 09:50:12 oak-gw06 kernel: 00000000000080d0 000000004605052a ffff880233cbb858 ffffffff8168662f Aug 1 09:50:12 oak-gw06 kernel: ffff880233cbb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 09:50:12 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880233cbb8b8 000000004605052a Aug 1 09:50:12 oak-gw06 kernel: Call Trace: Aug 1 09:50:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 09:50:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 09:50:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 09:50:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 09:50:12 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 09:50:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 09:50:12 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 09:50:12 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 09:50:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 09:50:12 oak-gw06 kernel: [] ? mlx4_ib_create_qp+0xfc/0x470 [mlx4_ib] Aug 1 09:50:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 09:50:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 09:50:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 09:50:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 09:50:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 09:50:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 09:50:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 09:50:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 09:50:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 09:50:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 09:50:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:50:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 09:50:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:50:12 oak-gw06 kernel: Mem-Info: Aug 1 09:50:12 oak-gw06 kernel: active_anon:28434 inactive_anon:41519 isolated_anon:0#012 active_file:582896 inactive_file:1780963 isolated_file:0#012 unevictable:0 dirty:10484 writeback:2855 unstable:0#012 slab_reclaimable:49518 slab_unreclaimable:943849#012 mapped:4938 shmem:39000 pagetables:1644 bounce:0#012 free:498812 free_pcp:1098 free_cma:0 Aug 1 09:50:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 09:50:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 09:50:12 oak-gw06 kernel: Node 0 DMA32 free:330996kB min:11976kB low:14968kB high:17964kB active_anon:18556kB inactive_anon:31248kB active_file:441088kB inactive_file:1271952kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:5184kB writeback:2268kB mapped:1992kB shmem:31276kB slab_reclaimable:36256kB slab_unreclaimable:663848kB kernel_stack:1024kB pagetables:1028kB unstable:0kB bounce:0kB free_pcp:1192kB local_pcp:72kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:50:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 09:50:12 oak-gw06 kernel: Node 0 Normal free:1733092kB min:55536kB low:69420kB high:83304kB active_anon:95180kB inactive_anon:134828kB active_file:1890496kB inactive_file:5760052kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:36580kB writeback:9496kB mapped:17760kB shmem:124724kB slab_reclaimable:161816kB slab_unreclaimable:3112068kB kernel_stack:4656kB pagetables:5548kB unstable:0kB bounce:0kB free_pcp:2496kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:50:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 09:50:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 09:50:12 oak-gw06 kernel: Node 0 DMA32: 6289*4kB (UEM) 938*8kB (UEM) 10016*16kB (UEM) 3791*32kB (UEM) 488*64kB (UEM) 7*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 346356kB Aug 1 09:50:12 oak-gw06 kernel: Node 0 Normal: 33029*4kB (UEM) 15634*8kB (UEM) 48558*16kB (UEM) 19084*32kB (UEM) 1906*64kB (UEM) 39*128kB (UEM) 3*256kB (UE) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1772548kB Aug 1 09:50:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 09:50:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 09:50:12 oak-gw06 kernel: 2110516 total pagecache pages Aug 1 09:50:12 oak-gw06 kernel: 0 pages in swap cache Aug 1 09:50:12 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 09:50:12 oak-gw06 kernel: Free swap = 4194300kB Aug 1 09:50:12 oak-gw06 kernel: Total swap = 4194300kB Aug 1 09:50:12 oak-gw06 kernel: 4194203 pages RAM Aug 1 09:50:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 09:50:12 oak-gw06 kernel: 127313 pages reserved Aug 1 09:50:12 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 09:50:12 oak-gw06 kernel: CPU: 3 PID: 11786 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 09:50:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 09:50:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 09:50:12 oak-gw06 kernel: 00000000000080d0 000000004605052a ffff880233cbb808 ffffffff8168662f Aug 1 09:50:12 oak-gw06 kernel: ffff880233cbb898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 09:50:12 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880233cbb868 000000004605052a Aug 1 09:50:12 oak-gw06 kernel: Call Trace: Aug 1 09:50:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 09:50:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 09:50:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 09:50:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 09:50:12 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 09:50:12 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 09:50:12 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 09:50:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 09:50:12 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 09:50:12 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 09:50:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 09:50:12 oak-gw06 kernel: [] ? mlx4_ib_create_qp+0xfc/0x470 [mlx4_ib] Aug 1 09:50:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 09:50:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 09:50:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 09:50:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 09:50:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 09:50:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 09:50:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 09:50:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 09:50:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 09:50:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 09:50:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:50:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 09:50:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:50:12 oak-gw06 kernel: Mem-Info: Aug 1 09:50:12 oak-gw06 kernel: active_anon:28434 inactive_anon:41519 isolated_anon:0#012 active_file:575181 inactive_file:1740999 isolated_file:0#012 unevictable:0 dirty:8415 writeback:3480 unstable:0#012 slab_reclaimable:49518 slab_unreclaimable:943439#012 mapped:4938 shmem:39000 pagetables:1644 bounce:0#012 free:550274 free_pcp:1576 free_cma:0 Aug 1 09:50:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 09:50:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 09:50:12 oak-gw06 kernel: Node 0 DMA32 free:367036kB min:11976kB low:14968kB high:17964kB active_anon:18556kB inactive_anon:31248kB active_file:435448kB inactive_file:1246384kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:5184kB writeback:1708kB mapped:1992kB shmem:31276kB slab_reclaimable:36256kB slab_unreclaimable:663848kB kernel_stack:1024kB pagetables:1028kB unstable:0kB bounce:0kB free_pcp:3132kB local_pcp:8kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:50:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 09:50:12 oak-gw06 kernel: Node 0 Normal free:1813080kB min:55536kB low:69420kB high:83304kB active_anon:95180kB inactive_anon:134828kB active_file:1865276kB inactive_file:5728072kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:32312kB writeback:11824kB mapped:17760kB shmem:124724kB slab_reclaimable:161816kB slab_unreclaimable:3109892kB kernel_stack:4656kB pagetables:5548kB unstable:0kB bounce:0kB free_pcp:3692kB local_pcp:56kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:50:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 09:50:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 09:50:12 oak-gw06 kernel: Node 0 DMA32: 5954*4kB (UEM) 3440*8kB (UEM) 10246*16kB (UEM) 3794*32kB (UEM) 488*64kB (UEM) 7*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 368808kB Aug 1 09:50:12 oak-gw06 kernel: Node 0 Normal: 29163*4kB (UEM) 22339*8kB (UEM) 49257*16kB (UEM) 19122*32kB (UEM) 1908*64kB (UEM) 39*128kB (UEM) 3*256kB (UE) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1823252kB Aug 1 09:50:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 09:50:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 09:50:12 oak-gw06 kernel: 2096638 total pagecache pages Aug 1 09:50:12 oak-gw06 kernel: 0 pages in swap cache Aug 1 09:50:12 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 09:50:12 oak-gw06 kernel: Free swap = 4194300kB Aug 1 09:50:12 oak-gw06 kernel: Total swap = 4194300kB Aug 1 09:50:12 oak-gw06 kernel: 4194203 pages RAM Aug 1 09:50:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 09:50:12 oak-gw06 kernel: 127313 pages reserved Aug 1 09:55:12 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 1 09:55:12 oak-gw06 kernel: CPU: 7 PID: 11800 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 09:55:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 09:55:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 09:55:12 oak-gw06 kernel: 00000000000080d0 00000000df7ffbe5 ffff88019e1d7858 ffffffff8168662f Aug 1 09:55:12 oak-gw06 kernel: ffff88019e1d78e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 09:55:12 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019e1d78b8 00000000df7ffbe5 Aug 1 09:55:12 oak-gw06 kernel: Call Trace: Aug 1 09:55:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 09:55:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 09:55:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 09:55:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 09:55:12 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 09:55:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 09:55:12 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 09:55:12 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 09:55:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 09:55:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 09:55:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 09:55:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 09:55:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 09:55:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 09:55:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 09:55:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 09:55:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 09:55:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 09:55:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 09:55:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:55:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 09:55:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:55:12 oak-gw06 kernel: Mem-Info: Aug 1 09:55:12 oak-gw06 kernel: active_anon:23252 inactive_anon:41519 isolated_anon:0#012 active_file:1066464 inactive_file:1184876 isolated_file:0#012 unevictable:0 dirty:2599 writeback:3929 unstable:0#012 slab_reclaimable:49008 slab_unreclaimable:941327#012 mapped:4959 shmem:39000 pagetables:1645 bounce:0#012 free:666195 free_pcp:1110 free_cma:0 Aug 1 09:55:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 09:55:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 09:55:12 oak-gw06 kernel: Node 0 DMA32 free:515512kB min:11976kB low:14968kB high:17964kB active_anon:14564kB inactive_anon:31248kB active_file:739632kB inactive_file:835096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1776kB writeback:1912kB mapped:1996kB shmem:31276kB slab_reclaimable:35900kB slab_unreclaimable:658832kB kernel_stack:1008kB pagetables:1104kB unstable:0kB bounce:0kB free_pcp:1188kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:55:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 09:55:12 oak-gw06 kernel: Node 0 Normal free:2130980kB min:55536kB low:69420kB high:83304kB active_anon:78444kB inactive_anon:134828kB active_file:3526224kB inactive_file:3906228kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8232kB writeback:11864kB mapped:17840kB shmem:124724kB slab_reclaimable:160132kB slab_unreclaimable:3106460kB kernel_stack:4672kB pagetables:5476kB unstable:0kB bounce:0kB free_pcp:3776kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:55:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 09:55:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 09:55:12 oak-gw06 kernel: Node 0 DMA32: 2090*4kB (UEM) 901*8kB (UEM) 13914*16kB (UEM) 7098*32kB (UEM) 746*64kB (UEM) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 514864kB Aug 1 09:55:12 oak-gw06 kernel: Node 0 Normal: 7850*4kB (UEM) 14221*8kB (UEM) 64169*16kB (UEM) 26052*32kB (UEM) 1814*64kB (UEM) 51*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2128416kB Aug 1 09:55:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 09:55:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 09:55:12 oak-gw06 kernel: 2109552 total pagecache pages Aug 1 09:55:12 oak-gw06 kernel: 0 pages in swap cache Aug 1 09:55:12 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 09:55:12 oak-gw06 kernel: Free swap = 4194300kB Aug 1 09:55:12 oak-gw06 kernel: Total swap = 4194300kB Aug 1 09:55:12 oak-gw06 kernel: 4194203 pages RAM Aug 1 09:55:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 09:55:12 oak-gw06 kernel: 127313 pages reserved Aug 1 09:55:12 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 1 09:55:12 oak-gw06 kernel: CPU: 7 PID: 11800 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 09:55:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 09:55:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 09:55:12 oak-gw06 kernel: 00000000000080d0 00000000df7ffbe5 ffff88019e1d7808 ffffffff8168662f Aug 1 09:55:12 oak-gw06 kernel: ffff88019e1d7898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 1 09:55:12 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019e1d7868 00000000df7ffbe5 Aug 1 09:55:12 oak-gw06 kernel: Call Trace: Aug 1 09:55:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 09:55:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 09:55:12 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 1 09:55:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 09:55:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 09:55:12 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 09:55:12 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 09:55:12 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 09:55:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 09:55:13 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 09:55:13 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 09:55:13 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 09:55:13 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 09:55:13 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 09:55:13 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 09:55:13 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 09:55:13 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 09:55:13 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 09:55:13 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 09:55:13 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 09:55:13 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 09:55:13 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 09:55:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:55:13 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 09:55:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 09:55:13 oak-gw06 kernel: Mem-Info: Aug 1 09:55:13 oak-gw06 kernel: active_anon:23346 inactive_anon:41519 isolated_anon:0#012 active_file:1066399 inactive_file:1190019 isolated_file:0#012 unevictable:0 dirty:2265 writeback:6106 unstable:0#012 slab_reclaimable:49008 slab_unreclaimable:941395#012 mapped:4959 shmem:39000 pagetables:1645 bounce:0#012 free:661426 free_pcp:760 free_cma:0 Aug 1 09:55:13 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 09:55:13 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 09:55:13 oak-gw06 kernel: Node 0 DMA32 free:512744kB min:11976kB low:14968kB high:17964kB active_anon:14940kB inactive_anon:31248kB active_file:739632kB inactive_file:838480kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1216kB writeback:2472kB mapped:1996kB shmem:31276kB slab_reclaimable:35900kB slab_unreclaimable:658832kB kernel_stack:1008kB pagetables:1104kB unstable:0kB bounce:0kB free_pcp:772kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:55:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 09:55:13 oak-gw06 kernel: Node 0 Normal free:2110892kB min:55536kB low:69420kB high:83304kB active_anon:78704kB inactive_anon:134828kB active_file:3525964kB inactive_file:3927028kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9008kB writeback:18848kB mapped:17840kB shmem:124724kB slab_reclaimable:160132kB slab_unreclaimable:3106732kB kernel_stack:4672kB pagetables:5476kB unstable:0kB bounce:0kB free_pcp:1788kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 09:55:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 09:55:13 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 09:55:13 oak-gw06 kernel: Node 0 DMA32: 1963*4kB (UEM) 904*8kB (UE) 13686*16kB (UEM) 7099*32kB (UEM) 746*64kB (UEM) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 510764kB Aug 1 09:55:13 oak-gw06 kernel: Node 0 Normal: 8563*4kB (UE) 11670*8kB (UEM) 64136*16kB (UEM) 26049*32kB (UEM) 1814*64kB (UEM) 51*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2110236kB Aug 1 09:55:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 09:55:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 09:55:13 oak-gw06 kernel: 2111955 total pagecache pages Aug 1 09:55:13 oak-gw06 kernel: 0 pages in swap cache Aug 1 09:55:13 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 09:55:13 oak-gw06 kernel: Free swap = 4194300kB Aug 1 09:55:13 oak-gw06 kernel: Total swap = 4194300kB Aug 1 09:55:13 oak-gw06 kernel: 4194203 pages RAM Aug 1 09:55:13 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 09:55:13 oak-gw06 kernel: 127313 pages reserved Aug 1 10:00:13 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 10:00:13 oak-gw06 kernel: CPU: 7 PID: 11786 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:00:13 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:00:13 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:00:13 oak-gw06 kernel: 00000000000080d0 000000004605052a ffff880233cbb858 ffffffff8168662f Aug 1 10:00:13 oak-gw06 kernel: ffff880233cbb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 10:00:13 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880233cbb8b8 000000004605052a Aug 1 10:00:13 oak-gw06 kernel: Call Trace: Aug 1 10:00:13 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:00:13 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:00:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:00:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:00:13 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 10:00:13 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 10:00:13 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:00:13 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:00:13 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:00:13 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:00:13 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:00:13 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:00:13 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:00:13 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:00:13 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:00:13 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:00:13 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:00:13 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:00:13 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:00:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:00:13 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:00:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:00:13 oak-gw06 kernel: Mem-Info: Aug 1 10:00:13 oak-gw06 kernel: active_anon:25545 inactive_anon:41519 isolated_anon:0#012 active_file:1006634 inactive_file:1280797 isolated_file:0#012 unevictable:0 dirty:2325 writeback:194 unstable:0#012 slab_reclaimable:49008 slab_unreclaimable:943066#012 mapped:4976 shmem:39000 pagetables:1650 bounce:0#012 free:626342 free_pcp:623 free_cma:0 Aug 1 10:00:13 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:00:13 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:00:13 oak-gw06 kernel: Node 0 DMA32 free:427040kB min:11976kB low:14968kB high:17964kB active_anon:15188kB inactive_anon:31248kB active_file:722344kB inactive_file:931096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2612kB writeback:416kB mapped:1992kB shmem:31276kB slab_reclaimable:35900kB slab_unreclaimable:661708kB kernel_stack:1040kB pagetables:1104kB unstable:0kB bounce:0kB free_pcp:1112kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:00:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:00:13 oak-gw06 kernel: Node 0 Normal free:2058324kB min:55536kB low:69420kB high:83304kB active_anon:86992kB inactive_anon:134828kB active_file:3304192kB inactive_file:4195964kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:7808kB writeback:748kB mapped:17912kB shmem:124724kB slab_reclaimable:160132kB slab_unreclaimable:3110540kB kernel_stack:4672kB pagetables:5496kB unstable:0kB bounce:0kB free_pcp:2708kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:00:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:00:13 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:00:13 oak-gw06 kernel: Node 0 DMA32: 1925*4kB (UEM) 5021*8kB (UEM) 9700*16kB (UEM) 5016*32kB (UEM) 932*64kB (UEM) 24*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 426556kB Aug 1 10:00:13 oak-gw06 kernel: Node 0 Normal: 8291*4kB (UEM) 17700*8kB (UEM) 49764*16kB (UEM) 26533*32kB (UEM) 3545*64kB (UEM) 80*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2057420kB Aug 1 10:00:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:00:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:00:13 oak-gw06 kernel: 2106311 total pagecache pages Aug 1 10:00:13 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:00:13 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:00:13 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:00:13 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:00:13 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:00:13 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:00:13 oak-gw06 kernel: 127313 pages reserved Aug 1 10:00:13 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 10:00:13 oak-gw06 kernel: CPU: 2 PID: 11786 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:00:13 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:00:13 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:00:13 oak-gw06 kernel: 00000000000080d0 000000004605052a ffff880233cbb808 ffffffff8168662f Aug 1 10:00:13 oak-gw06 kernel: ffff880233cbb898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 10:00:13 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880233cbb868 000000004605052a Aug 1 10:00:13 oak-gw06 kernel: Call Trace: Aug 1 10:00:13 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:00:13 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:00:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:00:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:00:13 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:00:13 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:00:13 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 10:00:13 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 10:00:13 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:00:13 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:00:13 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:00:13 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:00:13 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:00:13 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:00:13 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:00:13 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:00:13 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:00:13 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:00:13 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:00:13 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:00:13 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:00:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:00:13 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:00:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:00:13 oak-gw06 kernel: Mem-Info: Aug 1 10:00:13 oak-gw06 kernel: active_anon:25545 inactive_anon:41519 isolated_anon:0#012 active_file:1006634 inactive_file:1283029 isolated_file:0#012 unevictable:0 dirty:2368 writeback:679 unstable:0#012 slab_reclaimable:49008 slab_unreclaimable:943066#012 mapped:4976 shmem:39000 pagetables:1650 bounce:0#012 free:624259 free_pcp:535 free_cma:0 Aug 1 10:00:13 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:00:13 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:00:13 oak-gw06 kernel: Node 0 DMA32 free:428996kB min:11976kB low:14968kB high:17964kB active_anon:15188kB inactive_anon:31248kB active_file:722344kB inactive_file:931924kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1496kB writeback:308kB mapped:1992kB shmem:31276kB slab_reclaimable:35900kB slab_unreclaimable:661836kB kernel_stack:1040kB pagetables:1104kB unstable:0kB bounce:0kB free_pcp:680kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:00:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:00:13 oak-gw06 kernel: Node 0 Normal free:2050144kB min:55536kB low:69420kB high:83304kB active_anon:86992kB inactive_anon:134828kB active_file:3304192kB inactive_file:4202216kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:6280kB writeback:4304kB mapped:17948kB shmem:124724kB slab_reclaimable:160132kB slab_unreclaimable:3110412kB kernel_stack:4672kB pagetables:5496kB unstable:0kB bounce:0kB free_pcp:2784kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:00:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:00:13 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:00:13 oak-gw06 kernel: Node 0 DMA32: 2070*4kB (UEM) 4892*8kB (UEM) 9735*16kB (UEM) 5016*32kB (UEM) 932*64kB (UEM) 24*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 426664kB Aug 1 10:00:13 oak-gw06 kernel: Node 0 Normal: 8068*4kB (UEM) 16761*8kB (UEM) 49714*16kB (UEM) 26535*32kB (UEM) 3545*64kB (UEM) 80*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2048280kB Aug 1 10:00:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:00:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:00:13 oak-gw06 kernel: 2108627 total pagecache pages Aug 1 10:00:13 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:00:13 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:00:13 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:00:13 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:00:13 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:00:13 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:00:13 oak-gw06 kernel: 127313 pages reserved Aug 1 10:05:13 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 10:05:13 oak-gw06 kernel: CPU: 7 PID: 11786 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:05:13 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:05:13 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:05:13 oak-gw06 kernel: 00000000000080d0 000000004605052a ffff880233cbb858 ffffffff8168662f Aug 1 10:05:13 oak-gw06 kernel: ffff880233cbb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 10:05:13 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880233cbb8b8 000000004605052a Aug 1 10:05:13 oak-gw06 kernel: Call Trace: Aug 1 10:05:13 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:05:13 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:05:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:05:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:05:13 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 10:05:13 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 10:05:13 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:05:13 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:05:13 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:05:13 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:05:13 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:05:13 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:05:13 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:05:13 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:05:13 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:05:13 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:05:13 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:05:13 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:05:13 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:05:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:05:13 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:05:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:05:13 oak-gw06 kernel: Mem-Info: Aug 1 10:05:13 oak-gw06 kernel: active_anon:19425 inactive_anon:41519 isolated_anon:0#012 active_file:751800 inactive_file:1412422 isolated_file:0#012 unevictable:0 dirty:2183 writeback:2367 unstable:0#012 slab_reclaimable:49008 slab_unreclaimable:940662#012 mapped:4988 shmem:39000 pagetables:1486 bounce:0#012 free:757961 free_pcp:602 free_cma:0 Aug 1 10:05:13 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:05:13 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:05:13 oak-gw06 kernel: Node 0 DMA32 free:492480kB min:11976kB low:14968kB high:17964kB active_anon:13200kB inactive_anon:31248kB active_file:529544kB inactive_file:1064764kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:560kB writeback:1972kB mapped:1992kB shmem:31276kB slab_reclaimable:35900kB slab_unreclaimable:659472kB kernel_stack:1040kB pagetables:1064kB unstable:0kB bounce:0kB free_pcp:692kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:05:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:05:13 oak-gw06 kernel: Node 0 Normal free:2514712kB min:55536kB low:69420kB high:83304kB active_anon:64500kB inactive_anon:134828kB active_file:2477656kB inactive_file:4588796kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:628kB writeback:3616kB mapped:17960kB shmem:124724kB slab_reclaimable:160132kB slab_unreclaimable:3103160kB kernel_stack:4624kB pagetables:4880kB unstable:0kB bounce:0kB free_pcp:1120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:05:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:05:13 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:05:13 oak-gw06 kernel: Node 0 DMA32: 1835*4kB (UEM) 21005*8kB (UEM) 11350*16kB (UEM) 3057*32kB (UEM) 550*64kB (UEM) 18*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 492308kB Aug 1 10:05:13 oak-gw06 kernel: Node 0 Normal: 7421*4kB (UE) 104314*8kB (UEM) 51448*16kB (UEM) 21870*32kB (UEM) 1835*64kB (UEM) 17*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2506820kB Aug 1 10:05:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:05:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:05:13 oak-gw06 kernel: 2112080 total pagecache pages Aug 1 10:05:13 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:05:13 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:05:13 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:05:13 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:05:13 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:05:13 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:05:13 oak-gw06 kernel: 127313 pages reserved Aug 1 10:05:13 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 10:05:13 oak-gw06 kernel: CPU: 7 PID: 11786 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:05:13 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:05:13 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:05:13 oak-gw06 kernel: 00000000000080d0 000000004605052a ffff880233cbb808 ffffffff8168662f Aug 1 10:05:13 oak-gw06 kernel: ffff880233cbb898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 1 10:05:13 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880233cbb868 000000004605052a Aug 1 10:05:13 oak-gw06 kernel: Call Trace: Aug 1 10:05:13 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:05:13 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:05:13 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 1 10:05:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:05:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:05:13 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:05:13 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:05:13 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 10:05:13 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 10:05:13 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:05:13 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:05:13 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:05:13 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:05:13 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:05:13 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:05:13 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:05:13 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:05:13 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:05:13 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:05:13 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:05:13 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:05:13 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:05:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:05:13 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:05:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:05:13 oak-gw06 kernel: Mem-Info: Aug 1 10:05:13 oak-gw06 kernel: active_anon:19425 inactive_anon:41519 isolated_anon:0#012 active_file:751672 inactive_file:1416867 isolated_file:0#012 unevictable:0 dirty:454 writeback:216 unstable:0#012 slab_reclaimable:49008 slab_unreclaimable:940958#012 mapped:4992 shmem:39000 pagetables:1486 bounce:0#012 free:748167 free_pcp:370 free_cma:0 Aug 1 10:05:13 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:05:13 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:05:13 oak-gw06 kernel: Node 0 DMA32 free:487564kB min:11976kB low:14968kB high:17964kB active_anon:13200kB inactive_anon:31248kB active_file:529336kB inactive_file:1068148kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:560kB writeback:804kB mapped:1992kB shmem:31276kB slab_reclaimable:35900kB slab_unreclaimable:659584kB kernel_stack:1040kB pagetables:1064kB unstable:0kB bounce:0kB free_pcp:624kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:05:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:05:13 oak-gw06 kernel: Node 0 Normal free:2479332kB min:55536kB low:69420kB high:83304kB active_anon:65020kB inactive_anon:134828kB active_file:2477352kB inactive_file:4602816kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:1428kB writeback:2172kB mapped:17976kB shmem:124724kB slab_reclaimable:160132kB slab_unreclaimable:3104776kB kernel_stack:4640kB pagetables:4880kB unstable:0kB bounce:0kB free_pcp:1556kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:05:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:05:13 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:05:13 oak-gw06 kernel: Node 0 DMA32: 1561*4kB (UE) 20598*8kB (UEM) 11216*16kB (UEM) 3057*32kB (UEM) 550*64kB (UEM) 18*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 485812kB Aug 1 10:05:13 oak-gw06 kernel: Node 0 Normal: 6091*4kB (UE) 102773*8kB (UEM) 50978*16kB (UEM) 21868*32kB (UEM) 1835*64kB (UEM) 17*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2481588kB Aug 1 10:05:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:05:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:05:13 oak-gw06 kernel: 2096443 total pagecache pages Aug 1 10:05:13 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:05:13 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:05:13 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:05:13 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:05:13 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:05:13 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:05:13 oak-gw06 kernel: 127313 pages reserved Aug 1 10:10:14 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 1 10:10:14 oak-gw06 kernel: CPU: 6 PID: 11849 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:10:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:10:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:10:14 oak-gw06 kernel: 00000000000080d0 0000000001fdcceb ffff8801b146b858 ffffffff8168662f Aug 1 10:10:14 oak-gw06 kernel: ffff8801b146b8e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 1 10:10:14 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff8801b146b8e8 0000000001fdcceb Aug 1 10:10:14 oak-gw06 kernel: Call Trace: Aug 1 10:10:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:10:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:10:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:10:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:10:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 10:10:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 10:10:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:10:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:10:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:10:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:10:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:10:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:10:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:10:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:10:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:10:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:10:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:10:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:10:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:10:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:10:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:10:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:10:14 oak-gw06 kernel: Mem-Info: Aug 1 10:10:14 oak-gw06 kernel: active_anon:17374 inactive_anon:41519 isolated_anon:0#012 active_file:766225 inactive_file:1296469 isolated_file:6#012 unevictable:0 dirty:13 writeback:0 unstable:0#012 slab_reclaimable:49072 slab_unreclaimable:940399#012 mapped:5002 shmem:39000 pagetables:1408 bounce:0#012 free:861349 free_pcp:2060 free_cma:0 Aug 1 10:10:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:10:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:10:14 oak-gw06 kernel: Node 0 DMA32 free:569348kB min:11976kB low:14968kB high:17964kB active_anon:13112kB inactive_anon:31248kB active_file:535708kB inactive_file:978688kB unevictable:0kB isolated(anon):0kB isolated(file):24kB present:3129332kB managed:2884592kB mlocked:0kB dirty:12kB writeback:0kB mapped:1992kB shmem:31276kB slab_reclaimable:35964kB slab_unreclaimable:659168kB kernel_stack:1040kB pagetables:1044kB unstable:0kB bounce:0kB free_pcp:4124kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:10:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:10:14 oak-gw06 kernel: Node 0 Normal free:2852188kB min:55536kB low:69420kB high:83304kB active_anon:56904kB inactive_anon:134828kB active_file:2529192kB inactive_file:4215004kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:40kB writeback:0kB mapped:18016kB shmem:124724kB slab_reclaimable:160324kB slab_unreclaimable:3102412kB kernel_stack:4656kB pagetables:4588kB unstable:0kB bounce:0kB free_pcp:3852kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:10:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:10:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:10:14 oak-gw06 kernel: Node 0 DMA32: 5396*4kB (UEM) 29017*8kB (UEM) 11290*16kB (UEM) 3062*32kB (UEM) 543*64kB (UEM) 18*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 569400kB Aug 1 10:10:14 oak-gw06 kernel: Node 0 Normal: 37658*4kB (UEM) 130230*8kB (UEM) 52446*16kB (UEM) 21925*32kB (UEM) 1825*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2852312kB Aug 1 10:10:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:10:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:10:14 oak-gw06 kernel: 2103647 total pagecache pages Aug 1 10:10:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:10:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:10:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:10:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:10:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:10:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:10:14 oak-gw06 kernel: 127313 pages reserved Aug 1 10:10:14 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 1 10:10:14 oak-gw06 kernel: CPU: 6 PID: 11849 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:10:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:10:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:10:14 oak-gw06 kernel: 00000000000080d0 0000000001fdcceb ffff8801b146b808 ffffffff8168662f Aug 1 10:10:14 oak-gw06 kernel: ffff8801b146b898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 1 10:10:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801b146b868 0000000001fdcceb Aug 1 10:10:14 oak-gw06 kernel: Call Trace: Aug 1 10:10:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:10:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:10:14 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 1 10:10:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:10:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:10:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:10:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:10:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 10:10:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 10:10:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:10:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:10:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:10:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:10:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:10:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:10:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:10:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:10:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:10:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:10:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:10:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:10:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:10:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:10:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:10:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:10:14 oak-gw06 kernel: Mem-Info: Aug 1 10:10:14 oak-gw06 kernel: active_anon:17374 inactive_anon:41519 isolated_anon:0#012 active_file:766225 inactive_file:1302461 isolated_file:6#012 unevictable:0 dirty:13 writeback:0 unstable:0#012 slab_reclaimable:49072 slab_unreclaimable:940399#012 mapped:5002 shmem:39000 pagetables:1408 bounce:0#012 free:856756 free_pcp:217 free_cma:0 Aug 1 10:10:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:10:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:10:14 oak-gw06 kernel: Node 0 DMA32 free:569316kB min:11976kB low:14968kB high:17964kB active_anon:13112kB inactive_anon:31248kB active_file:535708kB inactive_file:981320kB unevictable:0kB isolated(anon):0kB isolated(file):24kB present:3129332kB managed:2884592kB mlocked:0kB dirty:12kB writeback:0kB mapped:1992kB shmem:31276kB slab_reclaimable:35964kB slab_unreclaimable:659168kB kernel_stack:1040kB pagetables:1044kB unstable:0kB bounce:0kB free_pcp:188kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:10:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:10:14 oak-gw06 kernel: Node 0 Normal free:2841816kB min:55536kB low:69420kB high:83304kB active_anon:56384kB inactive_anon:134828kB active_file:2529192kB inactive_file:4228524kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:40kB writeback:0kB mapped:18016kB shmem:124724kB slab_reclaimable:160324kB slab_unreclaimable:3102412kB kernel_stack:4656kB pagetables:4588kB unstable:0kB bounce:0kB free_pcp:680kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:10:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:10:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:10:14 oak-gw06 kernel: Node 0 DMA32: 5590*4kB (UEM) 29017*8kB (UEM) 11289*16kB (UEM) 3062*32kB (UEM) 543*64kB (UEM) 18*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 570160kB Aug 1 10:10:14 oak-gw06 kernel: Node 0 Normal: 31918*4kB (UEM) 130219*8kB (UEM) 52447*16kB (UEM) 21930*32kB (UEM) 1825*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2829440kB Aug 1 10:10:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:10:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:10:14 oak-gw06 kernel: 2111697 total pagecache pages Aug 1 10:10:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:10:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:10:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:10:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:10:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:10:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:10:14 oak-gw06 kernel: 127313 pages reserved Aug 1 10:15:13 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 10:15:13 oak-gw06 kernel: CPU: 7 PID: 11786 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:15:13 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:15:13 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:15:13 oak-gw06 kernel: 00000000000080d0 000000004605052a ffff880233cbb858 ffffffff8168662f Aug 1 10:15:13 oak-gw06 kernel: ffff880233cbb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 10:15:13 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880233cbb8b8 000000004605052a Aug 1 10:15:13 oak-gw06 kernel: Call Trace: Aug 1 10:15:13 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:15:13 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:15:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:15:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:15:13 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 10:15:13 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 10:15:13 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:15:13 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:15:13 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:15:13 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:15:13 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:15:13 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:15:13 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:15:13 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:15:13 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:15:13 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:15:13 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:15:13 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:15:13 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:15:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:15:13 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:15:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:15:13 oak-gw06 kernel: Mem-Info: Aug 1 10:15:13 oak-gw06 kernel: active_anon:17374 inactive_anon:41519 isolated_anon:0#012 active_file:793672 inactive_file:1277712 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:49008 slab_unreclaimable:940637#012 mapped:5009 shmem:39000 pagetables:1408 bounce:0#012 free:854152 free_pcp:189 free_cma:0 Aug 1 10:15:13 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:15:13 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:15:13 oak-gw06 kernel: Node 0 DMA32 free:562568kB min:11976kB low:14968kB high:17964kB active_anon:13112kB inactive_anon:31248kB active_file:555488kB inactive_file:970048kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:1992kB shmem:31276kB slab_reclaimable:35900kB slab_unreclaimable:659152kB kernel_stack:1024kB pagetables:1044kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:15:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:15:13 oak-gw06 kernel: Node 0 Normal free:2824088kB min:55536kB low:69420kB high:83304kB active_anon:56384kB inactive_anon:134828kB active_file:2619200kB inactive_file:4154840kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:18044kB shmem:124724kB slab_reclaimable:160132kB slab_unreclaimable:3103380kB kernel_stack:4656kB pagetables:4588kB unstable:0kB bounce:0kB free_pcp:832kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:15:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:15:13 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:15:13 oak-gw06 kernel: Node 0 DMA32: 2171*4kB (UEM) 27270*8kB (UEM) 11491*16kB (UEM) 3496*32kB (UEM) 558*64kB (UEM) 21*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 560972kB Aug 1 10:15:13 oak-gw06 kernel: Node 0 Normal: 9841*4kB (UEM) 127882*8kB (UEM) 54394*16kB (UEM) 23161*32kB (UEM) 2309*64kB (UEM) 23*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2824596kB Aug 1 10:15:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:15:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:15:13 oak-gw06 kernel: 2114299 total pagecache pages Aug 1 10:15:13 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:15:13 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:15:13 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:15:13 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:15:13 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:15:13 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:15:13 oak-gw06 kernel: 127313 pages reserved Aug 1 10:15:13 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 10:15:13 oak-gw06 kernel: CPU: 7 PID: 11786 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:15:13 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:15:13 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:15:13 oak-gw06 kernel: 00000000000080d0 000000004605052a ffff880233cbb808 ffffffff8168662f Aug 1 10:15:13 oak-gw06 kernel: ffff880233cbb898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 1 10:15:13 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880233cbb868 000000004605052a Aug 1 10:15:13 oak-gw06 kernel: Call Trace: Aug 1 10:15:13 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:15:13 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:15:13 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 1 10:15:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:15:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:15:13 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:15:13 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:15:13 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 10:15:13 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 10:15:13 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:15:13 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:15:13 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:15:13 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:15:13 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:15:13 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:15:13 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:15:13 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:15:13 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:15:13 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:15:13 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:15:13 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:15:13 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:15:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:15:13 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:15:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:15:13 oak-gw06 kernel: Mem-Info: Aug 1 10:15:13 oak-gw06 kernel: active_anon:17374 inactive_anon:41519 isolated_anon:0#012 active_file:793607 inactive_file:1280168 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:49008 slab_unreclaimable:940773#012 mapped:5009 shmem:39000 pagetables:1408 bounce:0#012 free:851108 free_pcp:998 free_cma:0 Aug 1 10:15:13 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:15:13 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:15:13 oak-gw06 kernel: Node 0 DMA32 free:564904kB min:11976kB low:14968kB high:17964kB active_anon:13112kB inactive_anon:31248kB active_file:555488kB inactive_file:965536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:1992kB shmem:31276kB slab_reclaimable:35900kB slab_unreclaimable:659152kB kernel_stack:1024kB pagetables:1044kB unstable:0kB bounce:0kB free_pcp:3200kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:15:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:15:13 oak-gw06 kernel: Node 0 Normal free:2846108kB min:55536kB low:69420kB high:83304kB active_anon:56904kB inactive_anon:134828kB active_file:2618940kB inactive_file:4129880kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:18044kB shmem:124724kB slab_reclaimable:160132kB slab_unreclaimable:3103924kB kernel_stack:4656kB pagetables:4588kB unstable:0kB bounce:0kB free_pcp:3276kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:15:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:15:13 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:15:13 oak-gw06 kernel: Node 0 DMA32: 3027*4kB (UEM) 27482*8kB (UEM) 11494*16kB (UEM) 3496*32kB (UEM) 558*64kB (UEM) 21*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 566140kB Aug 1 10:15:13 oak-gw06 kernel: Node 0 Normal: 13990*4kB (UEM) 128689*8kB (UEM) 54372*16kB (UEM) 23165*32kB (UEM) 2309*64kB (UEM) 23*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2847424kB Aug 1 10:15:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:15:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:15:13 oak-gw06 kernel: 2106077 total pagecache pages Aug 1 10:15:13 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:15:13 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:15:13 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:15:13 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:15:13 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:15:13 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:15:13 oak-gw06 kernel: 127313 pages reserved Aug 1 10:16:46 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:46 oak-gw06 kernel: swapper/4: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:46 oak-gw06 kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:46 oak-gw06 kernel: 0000000000104020 803b52a30d0a83c1 ffff88043fd039d8 ffffffff8168662f Aug 1 10:16:46 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:16:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4ed540 803b52a30d0a83c1 Aug 1 10:16:46 oak-gw06 kernel: Call Trace: Aug 1 10:16:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] ? enqueue_task_fair+0x208/0x6c0 Aug 1 10:16:46 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] ? ttwu_do_wakeup+0x19/0xd0 Aug 1 10:16:46 oak-gw06 kernel: [] ? ttwu_do_activate.constprop.90+0x5d/0x70 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:46 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:16:46 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:16:46 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:16:46 oak-gw06 kernel: swapper/4: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:46 oak-gw06 kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:46 oak-gw06 kernel: 0000000000104020 803b52a30d0a83c1 ffff88043fd039d8 ffffffff8168662f Aug 1 10:16:46 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:16:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4ed540 803b52a30d0a83c1 Aug 1 10:16:46 oak-gw06 kernel: Call Trace: Aug 1 10:16:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] ? enqueue_task_fair+0x208/0x6c0 Aug 1 10:16:46 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] ? ttwu_do_wakeup+0x19/0xd0 Aug 1 10:16:46 oak-gw06 kernel: [] ? ttwu_do_activate.constprop.90+0x5d/0x70 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:46 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:16:46 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:16:46 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:16:46 oak-gw06 kernel: swapper/4: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:46 oak-gw06 kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:46 oak-gw06 kernel: 0000000000104020 803b52a30d0a83c1 ffff88043fd039d8 ffffffff8168662f Aug 1 10:16:46 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:16:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4ed540 803b52a30d0a83c1 Aug 1 10:16:46 oak-gw06 kernel: Call Trace: Aug 1 10:16:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] ? enqueue_task_fair+0x208/0x6c0 Aug 1 10:16:46 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] ? ttwu_do_wakeup+0x19/0xd0 Aug 1 10:16:46 oak-gw06 kernel: [] ? ttwu_do_activate.constprop.90+0x5d/0x70 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:46 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:16:46 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:16:46 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:16:46 oak-gw06 kernel: swapper/4: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:46 oak-gw06 kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:46 oak-gw06 kernel: 0000000000104020 803b52a30d0a83c1 ffff88043fd039d8 ffffffff8168662f Aug 1 10:16:46 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:16:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4ed540 803b52a30d0a83c1 Aug 1 10:16:46 oak-gw06 kernel: Call Trace: Aug 1 10:16:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] ? enqueue_task_fair+0x208/0x6c0 Aug 1 10:16:46 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:46 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:16:46 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:16:46 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:16:46 oak-gw06 kernel: swapper/4: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:46 oak-gw06 kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:46 oak-gw06 kernel: 0000000000104020 803b52a30d0a83c1 ffff88043fd039d8 ffffffff8168662f Aug 1 10:16:46 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:16:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4ed540 803b52a30d0a83c1 Aug 1 10:16:46 oak-gw06 kernel: Call Trace: Aug 1 10:16:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] ? enqueue_task_fair+0x208/0x6c0 Aug 1 10:16:46 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:46 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:16:46 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:16:46 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:16:46 oak-gw06 kernel: swapper/4: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:46 oak-gw06 kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:46 oak-gw06 kernel: 0000000000104020 803b52a30d0a83c1 ffff88043fd039d8 ffffffff8168662f Aug 1 10:16:46 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:16:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4ed540 803b52a30d0a83c1 Aug 1 10:16:46 oak-gw06 kernel: Call Trace: Aug 1 10:16:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] ? enqueue_task_fair+0x208/0x6c0 Aug 1 10:16:46 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:46 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:16:46 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:16:46 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:16:46 oak-gw06 kernel: swapper/4: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:46 oak-gw06 kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:46 oak-gw06 kernel: 0000000000104020 803b52a30d0a83c1 ffff88043fd039d8 ffffffff8168662f Aug 1 10:16:46 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:16:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4ed540 803b52a30d0a83c1 Aug 1 10:16:46 oak-gw06 kernel: Call Trace: Aug 1 10:16:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] ? enqueue_task_fair+0x208/0x6c0 Aug 1 10:16:46 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:46 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:16:46 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:16:46 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:16:46 oak-gw06 kernel: swapper/4: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:46 oak-gw06 kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:46 oak-gw06 kernel: 0000000000104020 803b52a30d0a83c1 ffff88043fd039d8 ffffffff8168662f Aug 1 10:16:46 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:16:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4ed540 803b52a30d0a83c1 Aug 1 10:16:46 oak-gw06 kernel: Call Trace: Aug 1 10:16:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] ? enqueue_task_fair+0x208/0x6c0 Aug 1 10:16:46 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:46 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:16:46 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:16:46 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:16:46 oak-gw06 kernel: swapper/4: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:46 oak-gw06 kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:46 oak-gw06 kernel: 0000000000104020 803b52a30d0a83c1 ffff88043fd039d8 ffffffff8168662f Aug 1 10:16:46 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:16:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4ed540 803b52a30d0a83c1 Aug 1 10:16:46 oak-gw06 kernel: Call Trace: Aug 1 10:16:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] ? enqueue_task_fair+0x208/0x6c0 Aug 1 10:16:46 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:46 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:16:46 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:16:46 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:16:46 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:16:46 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:46 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:16:46 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:16:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa3e00 00000000210cf37f Aug 1 10:16:46 oak-gw06 kernel: Call Trace: Aug 1 10:16:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:46 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:16:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:16:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:46 oak-gw06 kernel: [] ? osc_object_prune+0xca/0x150 [osc] Aug 1 10:16:46 oak-gw06 kernel: [] cl_object_prune+0x55/0x110 [obdclass] Aug 1 10:16:46 oak-gw06 kernel: [] lov_delete_raid0+0xe0/0x400 [lov] Aug 1 10:16:46 oak-gw06 kernel: [] ? osc_io_init+0x81/0x140 [osc] Aug 1 10:16:46 oak-gw06 kernel: [] lov_object_delete+0x79/0x2a0 [lov] Aug 1 10:16:46 oak-gw06 kernel: [] lu_object_free.isra.31+0x9d/0x1a0 [obdclass] Aug 1 10:16:46 oak-gw06 kernel: [] lu_object_put+0xc2/0x3d0 [obdclass] Aug 1 10:16:46 oak-gw06 kernel: [] cl_object_put+0xe/0x10 [obdclass] Aug 1 10:16:46 oak-gw06 kernel: [] cl_inode_fini+0x75/0x220 [lustre] Aug 1 10:16:46 oak-gw06 kernel: [] ll_clear_inode+0x20c/0x820 [lustre] Aug 1 10:16:46 oak-gw06 kernel: [] ll_delete_inode+0x58/0x1c0 [lustre] Aug 1 10:16:46 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:16:46 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:16:46 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:16:46 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:16:46 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:16:46 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:16:46 oak-gw06 kernel: [] ? vmpressure+0x61/0x90 Aug 1 10:16:46 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:16:46 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:16:46 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:16:46 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:16:46 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:16:46 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:46 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:16:46 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: warn_alloc_failed: 325 callbacks suppressed Aug 1 10:16:57 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:57 oak-gw06 kernel: ptlrpcd_00_07: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:57 oak-gw06 kernel: CPU: 3 PID: 1769 Comm: ptlrpcd_00_07 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:57 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:57 oak-gw06 kernel: 0000000000104020 00000000e0e72661 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:16:57 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 0000000000000000 0000000000000002 Aug 1 10:16:57 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa7440 00000000e0e72661 Aug 1 10:16:57 oak-gw06 kernel: Call Trace: Aug 1 10:16:57 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:57 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] ? __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:57 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:57 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:57 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? __get_free_pages+0xe/0x50 Aug 1 10:16:57 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] ? swiotlb_unmap_page+0x9/0x10 Aug 1 10:16:57 oak-gw06 kernel: [] ? bnx2x_rx_int+0x279/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:57 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:57 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:57 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:57 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:57 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x51/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x3e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] vvp_page_discard+0xa2/0x160 [lustre] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_invoid+0x68/0x170 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] ? cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_discard+0x13/0x20 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] discard_pagevec+0x60/0xd0 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:16:57 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: ptlrpcd_00_07: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:57 oak-gw06 kernel: CPU: 3 PID: 1769 Comm: ptlrpcd_00_07 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:57 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:57 oak-gw06 kernel: 0000000000104020 00000000e0e72661 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:16:57 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 0000000000000000 0000000000000002 Aug 1 10:16:57 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa7440 00000000e0e72661 Aug 1 10:16:57 oak-gw06 kernel: Call Trace: Aug 1 10:16:57 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:57 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] ? __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:57 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:57 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:57 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] ? swiotlb_unmap_page+0x9/0x10 Aug 1 10:16:57 oak-gw06 kernel: [] ? bnx2x_rx_int+0x279/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:57 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:57 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:57 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:57 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:57 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x51/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x3e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] vvp_page_discard+0xa2/0x160 [lustre] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_invoid+0x68/0x170 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] ? cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_discard+0x13/0x20 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] discard_pagevec+0x60/0xd0 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:16:57 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: ptlrpcd_00_07: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:57 oak-gw06 kernel: CPU: 3 PID: 1769 Comm: ptlrpcd_00_07 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:57 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:57 oak-gw06 kernel: 0000000000104020 00000000e0e72661 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:16:57 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 0000000000000000 0000000000000002 Aug 1 10:16:57 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa7440 00000000e0e72661 Aug 1 10:16:57 oak-gw06 kernel: Call Trace: Aug 1 10:16:57 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:57 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] ? __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:57 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:57 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:57 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] ? swiotlb_unmap_page+0x9/0x10 Aug 1 10:16:57 oak-gw06 kernel: [] ? bnx2x_rx_int+0x279/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:57 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:57 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:57 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:57 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:57 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x51/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x3e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] vvp_page_discard+0xa2/0x160 [lustre] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_invoid+0x68/0x170 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] ? cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_discard+0x13/0x20 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] discard_pagevec+0x60/0xd0 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:16:57 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: ptlrpcd_00_07: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:57 oak-gw06 kernel: CPU: 3 PID: 1769 Comm: ptlrpcd_00_07 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:57 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:57 oak-gw06 kernel: 0000000000104020 00000000e0e72661 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:16:57 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 0000000000000000 0000000000000002 Aug 1 10:16:57 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa7440 00000000e0e72661 Aug 1 10:16:57 oak-gw06 kernel: Call Trace: Aug 1 10:16:57 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:57 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] ? __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:57 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:57 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:57 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] ? swiotlb_unmap_page+0x9/0x10 Aug 1 10:16:57 oak-gw06 kernel: [] ? bnx2x_rx_int+0x279/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:57 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:57 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:57 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:57 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:57 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x51/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x3e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] vvp_page_discard+0xa2/0x160 [lustre] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_invoid+0x68/0x170 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] ? cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_discard+0x13/0x20 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] discard_pagevec+0x60/0xd0 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:16:57 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: ptlrpcd_00_07: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:57 oak-gw06 kernel: CPU: 3 PID: 1769 Comm: ptlrpcd_00_07 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:57 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:57 oak-gw06 kernel: 0000000000104020 00000000e0e72661 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:16:57 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 0000000000000000 0000000000000002 Aug 1 10:16:57 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa7440 00000000e0e72661 Aug 1 10:16:57 oak-gw06 kernel: Call Trace: Aug 1 10:16:57 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:57 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] ? __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:57 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:57 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:57 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] ? swiotlb_unmap_page+0x9/0x10 Aug 1 10:16:57 oak-gw06 kernel: [] ? bnx2x_rx_int+0x279/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:57 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:57 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:57 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:57 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:57 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x51/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x3e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] vvp_page_discard+0xa2/0x160 [lustre] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_invoid+0x68/0x170 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] ? cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_discard+0x13/0x20 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] discard_pagevec+0x60/0xd0 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:16:57 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: ptlrpcd_00_07: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:57 oak-gw06 kernel: CPU: 3 PID: 1769 Comm: ptlrpcd_00_07 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:57 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:57 oak-gw06 kernel: 0000000000104020 00000000e0e72661 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:16:57 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 0000000000000000 0000000000000002 Aug 1 10:16:57 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa7440 00000000e0e72661 Aug 1 10:16:57 oak-gw06 kernel: Call Trace: Aug 1 10:16:57 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:57 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] ? __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:57 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:57 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:57 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] ? swiotlb_unmap_page+0x9/0x10 Aug 1 10:16:57 oak-gw06 kernel: [] ? bnx2x_rx_int+0x279/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:57 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:57 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:57 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:57 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:57 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x51/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x3e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] vvp_page_discard+0xa2/0x160 [lustre] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_invoid+0x68/0x170 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] ? cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_discard+0x13/0x20 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] discard_pagevec+0x60/0xd0 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:16:57 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: ptlrpcd_00_07: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:57 oak-gw06 kernel: CPU: 3 PID: 1769 Comm: ptlrpcd_00_07 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:57 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:57 oak-gw06 kernel: 0000000000104020 00000000e0e72661 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:16:57 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 0000000000000000 0000000000000002 Aug 1 10:16:57 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa7440 00000000e0e72661 Aug 1 10:16:57 oak-gw06 kernel: Call Trace: Aug 1 10:16:57 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:57 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] ? __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:57 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:57 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:57 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] ? swiotlb_unmap_page+0x9/0x10 Aug 1 10:16:57 oak-gw06 kernel: [] ? bnx2x_rx_int+0x279/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:57 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:57 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:57 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:57 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:57 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x51/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x3e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] vvp_page_discard+0xa2/0x160 [lustre] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_invoid+0x68/0x170 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] ? cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_discard+0x13/0x20 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] discard_pagevec+0x60/0xd0 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:16:57 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: ptlrpcd_00_07: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:57 oak-gw06 kernel: CPU: 3 PID: 1769 Comm: ptlrpcd_00_07 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:57 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:57 oak-gw06 kernel: 0000000000104020 00000000e0e72661 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:16:57 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 0000000000000000 0000000000000002 Aug 1 10:16:57 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa7440 00000000e0e72661 Aug 1 10:16:57 oak-gw06 kernel: Call Trace: Aug 1 10:16:57 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:57 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] ? __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:57 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:57 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:57 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] ? swiotlb_unmap_page+0x9/0x10 Aug 1 10:16:57 oak-gw06 kernel: [] ? bnx2x_rx_int+0x279/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:57 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:57 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:57 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:57 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:57 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x51/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? delete_from_page_cache+0x3e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] vvp_page_discard+0xa2/0x160 [lustre] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_invoid+0x68/0x170 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] ? cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] cl_page_discard+0x13/0x20 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] discard_pagevec+0x60/0xd0 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 1 10:16:57 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:16:57 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: CPU: 4 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:57 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:57 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fd039d8 ffffffff8168662f Aug 1 10:16:57 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff880210aefcf0 0000000000000020 Aug 1 10:16:57 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffffffff815ce55f 00000000210cf37f Aug 1 10:16:57 oak-gw06 kernel: Call Trace: Aug 1 10:16:57 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:57 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:57 oak-gw06 kernel: [] ? tcp_transmit_skb+0x4af/0x990 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] ? security_sock_rcv_skb+0x16/0x20 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:57 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:57 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:57 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:57 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:57 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:57 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:57 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:57 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:57 oak-gw06 kernel: [] ? cl_page_get_trust+0x5/0x50 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] ? cl_vmpage_page+0x3b/0x140 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] ll_releasepage+0x73/0x1a0 [lustre] Aug 1 10:16:57 oak-gw06 kernel: [] try_to_release_page+0x32/0x50 Aug 1 10:16:57 oak-gw06 kernel: [] shrink_page_list+0x950/0xb00 Aug 1 10:16:57 oak-gw06 kernel: [] shrink_inactive_list+0x1fa/0x630 Aug 1 10:16:57 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:16:57 oak-gw06 kernel: [] ? ldlm_cli_pool_shrink+0x72/0x100 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:16:57 oak-gw06 kernel: [] balance_pgdat+0x48c/0x5e0 Aug 1 10:16:57 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:16:57 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:16:57 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:16:57 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:16:57 oak-gw06 kernel: CPU: 4 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:16:57 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:16:57 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fd039d8 ffffffff8168662f Aug 1 10:16:57 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff880210aefcf0 0000000000000020 Aug 1 10:16:57 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffffffff815ce55f 00000000210cf37f Aug 1 10:16:57 oak-gw06 kernel: Call Trace: Aug 1 10:16:57 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:16:57 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:16:57 oak-gw06 kernel: [] ? tcp_transmit_skb+0x4af/0x990 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:16:57 oak-gw06 kernel: [] ? security_sock_rcv_skb+0x16/0x20 Aug 1 10:16:57 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:16:57 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:16:57 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:16:57 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:16:57 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:16:57 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:16:57 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:16:57 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:16:57 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:16:57 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:16:57 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:16:57 oak-gw06 kernel: [] ? cl_page_get_trust+0x5/0x50 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] ? cl_vmpage_page+0x3b/0x140 [obdclass] Aug 1 10:16:57 oak-gw06 kernel: [] ll_releasepage+0x73/0x1a0 [lustre] Aug 1 10:16:57 oak-gw06 kernel: [] try_to_release_page+0x32/0x50 Aug 1 10:16:57 oak-gw06 kernel: [] shrink_page_list+0x950/0xb00 Aug 1 10:16:57 oak-gw06 kernel: [] shrink_inactive_list+0x1fa/0x630 Aug 1 10:16:57 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:16:57 oak-gw06 kernel: [] ? ldlm_cli_pool_shrink+0x72/0x100 [ptlrpc] Aug 1 10:16:57 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:16:57 oak-gw06 kernel: [] balance_pgdat+0x48c/0x5e0 Aug 1 10:16:57 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:16:57 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:16:57 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:16:57 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:57 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:16:57 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:58 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:16:58 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:16:58 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:16:58 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:11 oak-gw06 kernel: warn_alloc_failed: 270 callbacks suppressed Aug 1 10:18:11 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:11 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:11 oak-gw06 kernel: CPU: 3 PID: 11862 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:11 oak-gw06 kernel: 0000000000104020 00000000794955ba ffff88043fcc39d8 ffffffff8168662f Aug 1 10:18:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 0000000000000000 0000000000000002 Aug 1 10:18:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa07c0 00000000794955ba Aug 1 10:18:11 oak-gw06 kernel: Call Trace: Aug 1 10:18:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] ? __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ? __get_free_pages+0xe/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] ? swiotlb_unmap_page+0x9/0x10 Aug 1 10:18:11 oak-gw06 kernel: [] ? bnx2x_rx_int+0x279/0x17b0 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:11 oak-gw06 kernel: [] ? shrink_page_list+0x3f5/0xb00 Aug 1 10:18:11 oak-gw06 kernel: [] ? shrink_page_list+0x3f5/0xb00 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_inactive_list+0x1fa/0x630 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:11 oak-gw06 kernel: [] ? wake_up_process+0x23/0x40 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:11 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:11 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:11 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:11 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ll_readahead+0xcbd/0x1660 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ? ldlm_lock_decref+0x36/0x80 [ptlrpc] Aug 1 10:18:11 oak-gw06 kernel: [] ? osc_io_fini+0x10/0x10 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] ll_readpage+0x3ba/0x6c0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] generic_file_aio_read+0x3cf/0x790 Aug 1 10:18:11 oak-gw06 kernel: [] vvp_io_read_start+0x4b4/0x5a0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_io_generic+0x570/0xb50 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_aio_read+0x347/0x3e0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 1 10:18:11 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:11 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:11 oak-gw06 kernel: CPU: 3 PID: 11862 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:11 oak-gw06 kernel: 0000000000104020 00000000794955ba ffff88043fcc39d8 ffffffff8168662f Aug 1 10:18:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:18:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000794955ba Aug 1 10:18:11 oak-gw06 kernel: Call Trace: Aug 1 10:18:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] ? update_curr+0x104/0x190 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:11 oak-gw06 kernel: [] ? __list_del_entry+0x29/0xd0 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:11 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ll_readahead+0xcbd/0x1660 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ? ldlm_lock_decref+0x36/0x80 [ptlrpc] Aug 1 10:18:11 oak-gw06 kernel: [] ? osc_io_fini+0x10/0x10 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] ll_readpage+0x3ba/0x6c0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] generic_file_aio_read+0x3cf/0x790 Aug 1 10:18:11 oak-gw06 kernel: [] vvp_io_read_start+0x4b4/0x5a0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_io_generic+0x570/0xb50 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_aio_read+0x347/0x3e0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 1 10:18:11 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:11 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:11 oak-gw06 kernel: CPU: 3 PID: 11862 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:11 oak-gw06 kernel: 0000000000104020 00000000794955ba ffff88043fcc39d8 ffffffff8168662f Aug 1 10:18:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:18:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000794955ba Aug 1 10:18:11 oak-gw06 kernel: Call Trace: Aug 1 10:18:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] ? update_curr+0x104/0x190 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:11 oak-gw06 kernel: [] ? __list_del_entry+0x29/0xd0 Aug 1 10:18:11 oak-gw06 kernel: [] __osc_lru_del+0x2f/0x80 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] osc_page_delete+0x115/0x4e0 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] ll_releasepage+0xee/0x1a0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] try_to_release_page+0x32/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_page_list+0x950/0xb00 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_inactive_list+0x1fa/0x630 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:11 oak-gw06 kernel: [] ? wake_up_process+0x23/0x40 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:11 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:11 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:11 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:11 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ll_readahead+0xcbd/0x1660 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ? ldlm_lock_decref+0x36/0x80 [ptlrpc] Aug 1 10:18:11 oak-gw06 kernel: [] ? osc_io_fini+0x10/0x10 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] ll_readpage+0x3ba/0x6c0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] generic_file_aio_read+0x3cf/0x790 Aug 1 10:18:11 oak-gw06 kernel: [] vvp_io_read_start+0x4b4/0x5a0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_io_generic+0x570/0xb50 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_aio_read+0x347/0x3e0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 1 10:18:11 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:11 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:11 oak-gw06 kernel: CPU: 3 PID: 11862 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:11 oak-gw06 kernel: 0000000000104020 00000000794955ba ffff88043fcc39d8 ffffffff8168662f Aug 1 10:18:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:18:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000794955ba Aug 1 10:18:11 oak-gw06 kernel: Call Trace: Aug 1 10:18:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] ? update_curr+0x104/0x190 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:11 oak-gw06 kernel: [] ? __list_del_entry+0x29/0xd0 Aug 1 10:18:11 oak-gw06 kernel: [] __osc_lru_del+0x2f/0x80 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] osc_page_delete+0x115/0x4e0 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] ll_releasepage+0xee/0x1a0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] try_to_release_page+0x32/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_page_list+0x950/0xb00 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:11 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ll_readahead+0xcbd/0x1660 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ? ldlm_lock_decref+0x36/0x80 [ptlrpc] Aug 1 10:18:11 oak-gw06 kernel: [] ? osc_io_fini+0x10/0x10 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] ll_readpage+0x3ba/0x6c0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] generic_file_aio_read+0x3cf/0x790 Aug 1 10:18:11 oak-gw06 kernel: [] vvp_io_read_start+0x4b4/0x5a0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_io_generic+0x570/0xb50 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_aio_read+0x347/0x3e0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 1 10:18:11 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:11 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:11 oak-gw06 kernel: CPU: 3 PID: 11862 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:11 oak-gw06 kernel: 0000000000104020 00000000794955ba ffff88043fcc39d8 ffffffff8168662f Aug 1 10:18:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:18:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000794955ba Aug 1 10:18:11 oak-gw06 kernel: Call Trace: Aug 1 10:18:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] ? update_curr+0x104/0x190 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:11 oak-gw06 kernel: [] ? __list_del_entry+0x29/0xd0 Aug 1 10:18:11 oak-gw06 kernel: [] __osc_lru_del+0x2f/0x80 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] osc_page_delete+0x115/0x4e0 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] ll_releasepage+0xee/0x1a0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] try_to_release_page+0x32/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_page_list+0x950/0xb00 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_inactive_list+0x1fa/0x630 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:11 oak-gw06 kernel: [] ? wake_up_process+0x23/0x40 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:11 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:11 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:11 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:11 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ll_readahead+0xcbd/0x1660 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ? ldlm_lock_decref+0x36/0x80 [ptlrpc] Aug 1 10:18:11 oak-gw06 kernel: [] ? osc_io_fini+0x10/0x10 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] ll_readpage+0x3ba/0x6c0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] generic_file_aio_read+0x3cf/0x790 Aug 1 10:18:11 oak-gw06 kernel: [] vvp_io_read_start+0x4b4/0x5a0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_io_generic+0x570/0xb50 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_aio_read+0x347/0x3e0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 1 10:18:11 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:11 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:11 oak-gw06 kernel: CPU: 3 PID: 11862 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:11 oak-gw06 kernel: 0000000000104020 00000000794955ba ffff88043fcc39d8 ffffffff8168662f Aug 1 10:18:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:18:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000794955ba Aug 1 10:18:11 oak-gw06 kernel: Call Trace: Aug 1 10:18:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] ? update_curr+0x104/0x190 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:11 oak-gw06 kernel: [] ? __list_del_entry+0x29/0xd0 Aug 1 10:18:11 oak-gw06 kernel: [] __osc_lru_del+0x2f/0x80 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] osc_page_delete+0x115/0x4e0 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] ll_releasepage+0xee/0x1a0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] try_to_release_page+0x32/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_page_list+0x950/0xb00 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_inactive_list+0x1fa/0x630 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:11 oak-gw06 kernel: [] ? wake_up_process+0x23/0x40 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:11 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:11 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:11 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:11 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ll_readahead+0xcbd/0x1660 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ? ldlm_lock_decref+0x36/0x80 [ptlrpc] Aug 1 10:18:11 oak-gw06 kernel: [] ? osc_io_fini+0x10/0x10 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] ll_readpage+0x3ba/0x6c0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] generic_file_aio_read+0x3cf/0x790 Aug 1 10:18:11 oak-gw06 kernel: [] vvp_io_read_start+0x4b4/0x5a0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_io_generic+0x570/0xb50 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_aio_read+0x347/0x3e0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 1 10:18:11 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:11 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:11 oak-gw06 kernel: CPU: 3 PID: 11862 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:11 oak-gw06 kernel: 0000000000104020 00000000794955ba ffff88043fcc39d8 ffffffff8168662f Aug 1 10:18:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:18:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000794955ba Aug 1 10:18:11 oak-gw06 kernel: Call Trace: Aug 1 10:18:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] ? update_curr+0x104/0x190 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:11 oak-gw06 kernel: [] ? __list_del_entry+0x29/0xd0 Aug 1 10:18:11 oak-gw06 kernel: [] __osc_lru_del+0x2f/0x80 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] osc_page_delete+0x115/0x4e0 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] ll_releasepage+0xee/0x1a0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] try_to_release_page+0x32/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_page_list+0x950/0xb00 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_inactive_list+0x1fa/0x630 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:11 oak-gw06 kernel: [] ? wake_up_process+0x23/0x40 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:11 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:11 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:11 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:11 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ll_readahead+0xcbd/0x1660 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ? ldlm_lock_decref+0x36/0x80 [ptlrpc] Aug 1 10:18:11 oak-gw06 kernel: [] ? osc_io_fini+0x10/0x10 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] ll_readpage+0x3ba/0x6c0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] generic_file_aio_read+0x3cf/0x790 Aug 1 10:18:11 oak-gw06 kernel: [] vvp_io_read_start+0x4b4/0x5a0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_io_generic+0x570/0xb50 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_aio_read+0x347/0x3e0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 1 10:18:11 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:11 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:11 oak-gw06 kernel: CPU: 3 PID: 11862 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:11 oak-gw06 kernel: 0000000000104020 00000000794955ba ffff88043fcc39d8 ffffffff8168662f Aug 1 10:18:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88019832c050 ffff880415b500b8 Aug 1 10:18:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa07c0 00000000794955ba Aug 1 10:18:11 oak-gw06 kernel: Call Trace: Aug 1 10:18:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:11 oak-gw06 kernel: [] ? shrink_page_list+0x4cc/0xb00 Aug 1 10:18:11 oak-gw06 kernel: [] ? shrink_page_list+0x4c4/0xb00 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_inactive_list+0x1fa/0x630 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:11 oak-gw06 kernel: [] ? wake_up_process+0x23/0x40 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:11 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:11 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:11 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:11 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ll_readahead+0xcbd/0x1660 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ? ldlm_lock_decref+0x36/0x80 [ptlrpc] Aug 1 10:18:11 oak-gw06 kernel: [] ? osc_io_fini+0x10/0x10 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] ll_readpage+0x3ba/0x6c0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] generic_file_aio_read+0x3cf/0x790 Aug 1 10:18:11 oak-gw06 kernel: [] vvp_io_read_start+0x4b4/0x5a0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_io_generic+0x570/0xb50 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_aio_read+0x347/0x3e0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 1 10:18:11 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:11 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:11 oak-gw06 kernel: CPU: 3 PID: 11862 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:11 oak-gw06 kernel: 0000000000104020 00000000794955ba ffff88043fcc39d8 ffffffff8168662f Aug 1 10:18:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88019832c050 ffff880415b500b8 Aug 1 10:18:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa07c0 00000000794955ba Aug 1 10:18:11 oak-gw06 kernel: Call Trace: Aug 1 10:18:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:11 oak-gw06 kernel: [] ? shrink_page_list+0x4cc/0xb00 Aug 1 10:18:11 oak-gw06 kernel: [] ? shrink_page_list+0x4c4/0xb00 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_inactive_list+0x1fa/0x630 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:11 oak-gw06 kernel: [] ? wake_up_process+0x23/0x40 Aug 1 10:18:11 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:11 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:11 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:11 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:11 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] ll_readahead+0xcbd/0x1660 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ? ldlm_lock_decref+0x36/0x80 [ptlrpc] Aug 1 10:18:11 oak-gw06 kernel: [] ? osc_io_fini+0x10/0x10 [osc] Aug 1 10:18:11 oak-gw06 kernel: [] ll_readpage+0x3ba/0x6c0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] generic_file_aio_read+0x3cf/0x790 Aug 1 10:18:11 oak-gw06 kernel: [] vvp_io_read_start+0x4b4/0x5a0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_io_generic+0x570/0xb50 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_aio_read+0x347/0x3e0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 1 10:18:11 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 1 10:18:11 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:11 oak-gw06 kernel: CPU: 4 PID: 11863 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:11 oak-gw06 kernel: 0000000000104020 0000000003744db4 ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:11 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:18:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e8f80 0000000003744db4 Aug 1 10:18:11 oak-gw06 kernel: Call Trace: Aug 1 10:18:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:11 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:18:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:11 oak-gw06 kernel: [] ? sysret_audit+0x17/0x21 Aug 1 10:18:17 oak-gw06 kernel: warn_alloc_failed: 102 callbacks suppressed Aug 1 10:18:17 oak-gw06 kernel: ptlrpcd_00_06: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:17 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:17 oak-gw06 kernel: CPU: 4 PID: 11860 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:17 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:17 oak-gw06 kernel: 0000000000104020 000000008f78d2e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:17 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:18:17 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9f00 000000008f78d2e5 Aug 1 10:18:17 oak-gw06 kernel: Call Trace: Aug 1 10:18:17 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:17 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:17 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:17 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:17 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:17 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:17 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:17 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:17 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:18 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:18 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:18 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:18 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:18 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:18 oak-gw06 kernel: [] ? shrink_inactive_list+0x15d/0x630 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:18 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:18 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:18 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:18 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ll_write_begin+0xa1/0x830 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Aug 1 10:18:18 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:18:18 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:18:18 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:18:18 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:18:18 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:18 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:18 oak-gw06 kernel: CPU: 4 PID: 11860 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:18 oak-gw06 kernel: 0000000000104020 000000008f78d2e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:18 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:18:18 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9f00 000000008f78d2e5 Aug 1 10:18:18 oak-gw06 kernel: Call Trace: Aug 1 10:18:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:18 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:18 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:18 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:18 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:18 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:18 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:18 oak-gw06 kernel: [] ? shrink_inactive_list+0x15d/0x630 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:18 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:18 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:18 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:18 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ll_write_begin+0xa1/0x830 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Aug 1 10:18:18 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:18:18 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:18:18 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:18:18 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:18:18 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:18 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:18 oak-gw06 kernel: CPU: 4 PID: 11860 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:18 oak-gw06 kernel: 0000000000104020 000000008f78d2e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:18 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:18:18 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4ec5c0 000000008f78d2e5 Aug 1 10:18:18 oak-gw06 kernel: Call Trace: Aug 1 10:18:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:18 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:18 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:18 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:18 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:18 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:18 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:18 oak-gw06 kernel: [] ? shrink_inactive_list+0x15d/0x630 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:18 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:18 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:18 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:18 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ll_write_begin+0xa1/0x830 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Aug 1 10:18:18 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:18:18 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:18:18 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:18:18 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:18:18 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:18 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:18 oak-gw06 kernel: CPU: 4 PID: 11860 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:18 oak-gw06 kernel: 0000000000104020 000000008f78d2e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:18 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:18:18 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4ec5c0 000000008f78d2e5 Aug 1 10:18:18 oak-gw06 kernel: Call Trace: Aug 1 10:18:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:18 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:18 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:18 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:18 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:18 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:18 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:18 oak-gw06 kernel: [] ? shrink_inactive_list+0x15d/0x630 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:18 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:18 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:18 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:18 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ll_write_begin+0xa1/0x830 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Aug 1 10:18:18 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:18:18 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:18:18 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:18:18 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:18:18 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:18 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:18 oak-gw06 kernel: CPU: 4 PID: 11860 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:18 oak-gw06 kernel: 0000000000104020 000000008f78d2e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:18 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:18:18 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4ec5c0 000000008f78d2e5 Aug 1 10:18:18 oak-gw06 kernel: Call Trace: Aug 1 10:18:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:18 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:18 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:18 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:18 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:18 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:18 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:18 oak-gw06 kernel: [] ? shrink_inactive_list+0x15d/0x630 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:18 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:18 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:18 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:18 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ll_write_begin+0xa1/0x830 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Aug 1 10:18:18 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:18:18 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:18:18 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:18:18 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:18:18 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:18 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:18 oak-gw06 kernel: CPU: 4 PID: 11860 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:18 oak-gw06 kernel: 0000000000104020 000000008f78d2e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:18 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:18:18 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4ec5c0 000000008f78d2e5 Aug 1 10:18:18 oak-gw06 kernel: Call Trace: Aug 1 10:18:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:18 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:18 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:18 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:18 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:18 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:18 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:18 oak-gw06 kernel: [] ? shrink_inactive_list+0x15d/0x630 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:18 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:18 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:18 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:18 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ll_write_begin+0xa1/0x830 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Aug 1 10:18:18 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:18:18 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:18:18 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:18:18 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:18:18 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:18 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:18 oak-gw06 kernel: CPU: 4 PID: 11860 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:18 oak-gw06 kernel: 0000000000104020 000000008f78d2e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:18 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 0000000000000002 0000000000000002 Aug 1 10:18:18 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffffffff8119494f 000000008f78d2e5 Aug 1 10:18:18 oak-gw06 kernel: Call Trace: Aug 1 10:18:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] ? wakeup_kswapd+0x3f/0x140 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:18 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:18 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:18 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:18 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:18 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:18 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:18 oak-gw06 kernel: [] ? shrink_inactive_list+0x15d/0x630 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:18 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:18 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:18 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:18 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ll_write_begin+0xa1/0x830 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Aug 1 10:18:18 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:18:18 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:18:18 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:18:18 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:18:18 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:18 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:18 oak-gw06 kernel: CPU: 4 PID: 11860 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:18 oak-gw06 kernel: 0000000000104020 000000008f78d2e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:18 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 0000000000000002 0000000000000002 Aug 1 10:18:18 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffffffff8119494f 000000008f78d2e5 Aug 1 10:18:18 oak-gw06 kernel: Call Trace: Aug 1 10:18:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] ? wakeup_kswapd+0x3f/0x140 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:18 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:18 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:18 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:18 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:18 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:18 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:18 oak-gw06 kernel: [] ? shrink_inactive_list+0x15d/0x630 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:18 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:18 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:18 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:18 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ll_write_begin+0xa1/0x830 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Aug 1 10:18:18 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:18:18 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:18:18 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:18:18 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:18:18 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:18 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:18 oak-gw06 kernel: CPU: 4 PID: 11860 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:18 oak-gw06 kernel: 0000000000104020 000000008f78d2e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:18 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 0000000000000002 0000000000000002 Aug 1 10:18:18 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffffffff8119494f 000000008f78d2e5 Aug 1 10:18:18 oak-gw06 kernel: Call Trace: Aug 1 10:18:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] ? wakeup_kswapd+0x3f/0x140 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:18 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:18 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:18 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:18 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:18 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:18 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:18 oak-gw06 kernel: [] ? shrink_inactive_list+0x15d/0x630 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:18:18 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:18:18 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:18:18 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 1 10:18:18 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] ? radix_tree_gang_lookup_tag_slot+0xf0/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 1 10:18:18 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] ll_write_begin+0xa1/0x830 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Aug 1 10:18:18 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:18:18 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:18:18 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:18:18 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:18:18 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:18:18 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:18:18 oak-gw06 kernel: CPU: 3 PID: 1768 Comm: ptlrpcd_00_06 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:18 oak-gw06 kernel: 0000000000104020 00000000e741dcae ffff88043fcc39d8 ffffffff8168662f Aug 1 10:18:18 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:18:18 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa64c0 00000000e741dcae Aug 1 10:18:18 oak-gw06 kernel: Call Trace: Aug 1 10:18:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:18 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:18 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:18 oak-gw06 kernel: [] ? sched_clock+0x9/0x10 Aug 1 10:18:18 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:18 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:18 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:18 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:18 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:18 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:18 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:18 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:18 oak-gw06 kernel: [] ? loop_64+0x4/0x78 [crc32_pclmul] Aug 1 10:18:18 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:18:18 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:18:18 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:18:18 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:18:18 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:18:18 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:18:18 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:18:18 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:18:18 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 1 10:18:18 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 1 10:18:18 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:18:18 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 1 10:18:18 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:18:18 oak-gw06 kernel: [] ? null_free_reqbuf+0x124/0x2e0 [ptlrpc] Aug 1 10:18:18 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:18:18 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:18:18 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 1 10:18:18 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:18:18 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:18:18 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:18:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:18 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:18:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: warn_alloc_failed: 1690 callbacks suppressed Aug 1 10:18:26 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:26 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:26 oak-gw06 kernel: CPU: 4 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:26 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:26 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:26 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88041e860050 ffff880415b500b8 Aug 1 10:18:26 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e87c0 00000000210cf37f Aug 1 10:18:26 oak-gw06 kernel: Call Trace: Aug 1 10:18:26 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:26 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:26 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:26 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:26 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:26 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:26 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:26 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:26 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:26 oak-gw06 kernel: [] ? __radix_tree_lookup+0x79/0xb0 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete_item+0x36/0x100 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete+0xb/0x10 Aug 1 10:18:26 oak-gw06 kernel: [] osc_page_delete+0x202/0x4e0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_lru_shrink+0x3f7/0x750 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] ? cl_env_get+0x1bb/0x270 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink_scan+0x143/0x190 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink+0x36/0x60 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:18:26 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:18:26 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:26 oak-gw06 kernel: CPU: 4 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:26 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:26 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:26 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88041e860050 ffff880415b500b8 Aug 1 10:18:26 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e87c0 00000000210cf37f Aug 1 10:18:26 oak-gw06 kernel: Call Trace: Aug 1 10:18:26 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:26 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:26 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:26 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:26 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:26 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:26 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:26 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:26 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:26 oak-gw06 kernel: [] ? __radix_tree_lookup+0x79/0xb0 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete_item+0x36/0x100 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete+0xb/0x10 Aug 1 10:18:26 oak-gw06 kernel: [] osc_page_delete+0x202/0x4e0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_lru_shrink+0x3f7/0x750 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] ? cl_env_get+0x1bb/0x270 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink_scan+0x143/0x190 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink+0x36/0x60 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:18:26 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:18:26 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:26 oak-gw06 kernel: CPU: 4 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:26 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:26 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:26 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88041e860050 ffff880415b500b8 Aug 1 10:18:26 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e87c0 00000000210cf37f Aug 1 10:18:26 oak-gw06 kernel: Call Trace: Aug 1 10:18:26 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:26 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:26 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:26 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:26 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:26 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:26 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:26 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:26 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:26 oak-gw06 kernel: [] ? __radix_tree_lookup+0x79/0xb0 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete_item+0x36/0x100 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete+0xb/0x10 Aug 1 10:18:26 oak-gw06 kernel: [] osc_page_delete+0x202/0x4e0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_lru_shrink+0x3f7/0x750 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] ? cl_env_get+0x1bb/0x270 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink_scan+0x143/0x190 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink+0x36/0x60 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:18:26 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:18:26 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:26 oak-gw06 kernel: CPU: 4 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:26 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:26 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:26 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88041e860050 ffff880415b500b8 Aug 1 10:18:26 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e87c0 00000000210cf37f Aug 1 10:18:26 oak-gw06 kernel: Call Trace: Aug 1 10:18:26 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:26 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:26 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:26 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:26 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:26 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:26 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:26 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:26 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:26 oak-gw06 kernel: [] ? __radix_tree_lookup+0x79/0xb0 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete_item+0x36/0x100 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete+0xb/0x10 Aug 1 10:18:26 oak-gw06 kernel: [] osc_page_delete+0x202/0x4e0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_lru_shrink+0x3f7/0x750 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] ? cl_env_get+0x1bb/0x270 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink_scan+0x143/0x190 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink+0x36/0x60 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:18:26 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:18:26 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:26 oak-gw06 kernel: CPU: 4 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:26 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:26 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:26 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88041e860050 ffff880415b500b8 Aug 1 10:18:26 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e87c0 00000000210cf37f Aug 1 10:18:26 oak-gw06 kernel: Call Trace: Aug 1 10:18:26 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:26 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:26 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:26 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:26 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:26 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:26 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:26 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:26 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:26 oak-gw06 kernel: [] ? __radix_tree_lookup+0x79/0xb0 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete_item+0x36/0x100 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete+0xb/0x10 Aug 1 10:18:26 oak-gw06 kernel: [] osc_page_delete+0x202/0x4e0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_lru_shrink+0x3f7/0x750 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] ? cl_env_get+0x1bb/0x270 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink_scan+0x143/0x190 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink+0x36/0x60 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:18:26 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:18:26 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:26 oak-gw06 kernel: CPU: 4 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:26 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:26 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:26 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88041e860050 ffff880415b500b8 Aug 1 10:18:26 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e87c0 00000000210cf37f Aug 1 10:18:26 oak-gw06 kernel: Call Trace: Aug 1 10:18:26 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:26 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:26 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:26 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:26 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:26 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:26 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:26 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:26 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:26 oak-gw06 kernel: [] ? __radix_tree_lookup+0x79/0xb0 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete_item+0x36/0x100 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete+0xb/0x10 Aug 1 10:18:26 oak-gw06 kernel: [] osc_page_delete+0x202/0x4e0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_lru_shrink+0x3f7/0x750 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] ? cl_env_get+0x1bb/0x270 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink_scan+0x143/0x190 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink+0x36/0x60 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:18:26 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:18:26 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:26 oak-gw06 kernel: CPU: 4 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:26 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:26 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:26 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88041e860050 ffff880415b500b8 Aug 1 10:18:26 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e87c0 00000000210cf37f Aug 1 10:18:26 oak-gw06 kernel: Call Trace: Aug 1 10:18:26 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:26 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:26 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:26 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:26 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:26 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:26 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:26 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:26 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:26 oak-gw06 kernel: [] ? __radix_tree_lookup+0x79/0xb0 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete_item+0x36/0x100 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete+0xb/0x10 Aug 1 10:18:26 oak-gw06 kernel: [] osc_page_delete+0x202/0x4e0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_lru_shrink+0x3f7/0x750 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] ? cl_env_get+0x1bb/0x270 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink_scan+0x143/0x190 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink+0x36/0x60 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:18:26 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:18:26 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:26 oak-gw06 kernel: CPU: 4 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:26 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:26 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:26 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88041e860050 ffff880415b500b8 Aug 1 10:18:26 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e87c0 00000000210cf37f Aug 1 10:18:26 oak-gw06 kernel: Call Trace: Aug 1 10:18:26 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:26 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:26 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:26 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:26 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:26 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:26 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:26 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:26 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:26 oak-gw06 kernel: [] ? __radix_tree_lookup+0x79/0xb0 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete_item+0x36/0x100 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete+0xb/0x10 Aug 1 10:18:26 oak-gw06 kernel: [] osc_page_delete+0x202/0x4e0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_lru_shrink+0x3f7/0x750 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] ? cl_env_get+0x1bb/0x270 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink_scan+0x143/0x190 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink+0x36/0x60 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:18:26 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:18:26 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:18:26 oak-gw06 kernel: CPU: 4 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:26 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:26 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fd039d8 ffffffff8168662f Aug 1 10:18:26 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88041e860050 ffff880415b500b8 Aug 1 10:18:26 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e87c0 00000000210cf37f Aug 1 10:18:26 oak-gw06 kernel: Call Trace: Aug 1 10:18:26 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:26 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:26 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:26 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:26 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:26 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:26 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:26 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:26 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:26 oak-gw06 kernel: [] ? __radix_tree_lookup+0x79/0xb0 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete_item+0x36/0x100 Aug 1 10:18:26 oak-gw06 kernel: [] radix_tree_delete+0xb/0x10 Aug 1 10:18:26 oak-gw06 kernel: [] osc_page_delete+0x202/0x4e0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_lru_shrink+0x3f7/0x750 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] ? cl_env_get+0x1bb/0x270 [obdclass] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink_scan+0x143/0x190 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] osc_cache_shrink+0x36/0x60 [osc] Aug 1 10:18:26 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:18:26 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:18:26 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:18:26 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:18:26 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:18:26 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:18:26 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:18:26 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:18:26 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8803bf3d0050 ffff880415b500b8 Aug 1 10:18:26 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa7440 a5232eaa14f210cb Aug 1 10:18:26 oak-gw06 kernel: Call Trace: Aug 1 10:18:26 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:18:26 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:18:26 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:18:26 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:18:26 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:18:26 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] ? napi_gro_complete+0x7d/0x100 Aug 1 10:18:26 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:18:26 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:18:26 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:18:26 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:18:26 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:18:26 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:18:26 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:18:26 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:18:26 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:18:26 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:18:26 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:18:26 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:19:35 oak-gw06 kernel: warn_alloc_failed: 54 callbacks suppressed Aug 1 10:19:35 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:35 oak-gw06 kernel: CPU: 4 PID: 11862 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:35 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:35 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:35 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:35 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:35 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:19:35 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000282396d3 Aug 1 10:19:35 oak-gw06 kernel: Call Trace: Aug 1 10:19:35 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:35 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:35 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:35 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:35 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:35 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:35 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:35 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:35 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:35 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:35 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:19:35 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:35 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:35 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:35 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:35 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:19:35 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000282396d3 Aug 1 10:19:35 oak-gw06 kernel: Call Trace: Aug 1 10:19:35 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:35 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:35 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:35 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:35 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:35 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:35 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:35 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:35 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:35 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:35 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:19:35 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:35 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:35 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:35 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:35 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:19:35 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000282396d3 Aug 1 10:19:35 oak-gw06 kernel: Call Trace: Aug 1 10:19:35 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:35 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:35 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:35 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:35 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:35 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:35 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:35 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:35 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:35 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:35 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:19:35 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:35 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:35 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:35 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:35 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:19:35 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000282396d3 Aug 1 10:19:35 oak-gw06 kernel: Call Trace: Aug 1 10:19:35 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:35 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:35 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:35 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:35 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:35 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:35 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:35 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:35 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:35 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:35 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:19:35 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:35 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:35 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:35 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:35 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:19:35 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000282396d3 Aug 1 10:19:35 oak-gw06 kernel: Call Trace: Aug 1 10:19:35 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:35 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:35 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:35 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:35 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:35 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:35 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:35 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:35 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:35 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:35 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:19:35 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:35 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:35 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:35 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:35 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:19:35 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000282396d3 Aug 1 10:19:35 oak-gw06 kernel: Call Trace: Aug 1 10:19:35 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:35 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:35 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:35 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:35 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:35 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:35 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:35 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:35 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:35 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:35 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:19:35 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:35 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:35 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:35 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:35 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:19:35 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000282396d3 Aug 1 10:19:35 oak-gw06 kernel: Call Trace: Aug 1 10:19:35 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:35 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:35 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:35 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:35 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:35 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:35 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:35 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:35 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:35 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:35 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:19:35 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:35 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:35 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:35 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:35 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:19:35 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000282396d3 Aug 1 10:19:35 oak-gw06 kernel: Call Trace: Aug 1 10:19:35 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:35 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:35 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:35 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:35 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:35 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:35 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:35 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:35 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:35 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:35 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:19:35 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:35 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:35 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:35 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:35 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:19:35 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000282396d3 Aug 1 10:19:35 oak-gw06 kernel: Call Trace: Aug 1 10:19:35 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:35 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:35 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:35 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:35 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:35 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:35 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:35 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:35 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:35 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:35 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:19:35 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:35 oak-gw06 kernel: 0000000000104020 00000000794955ba ffff88043fd039d8 ffffffff8168662f Aug 1 10:19:35 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 0000000000000010 0000000000000000 Aug 1 10:19:35 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9f00 00000000794955ba Aug 1 10:19:35 oak-gw06 kernel: Call Trace: Aug 1 10:19:35 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:35 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:35 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:35 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:35 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:35 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:35 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:35 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:35 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:35 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:35 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:35 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:35 oak-gw06 kernel: [] ? cl_page_assume+0x77/0x200 [obdclass] Aug 1 10:19:35 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:19:35 oak-gw06 kernel: [] ? iov_iter_copy_from_user_atomic+0x6a/0xa0 Aug 1 10:19:35 oak-gw06 kernel: [] generic_file_buffered_write+0x155/0x2a0 Aug 1 10:19:35 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:19:35 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:19:35 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:19:35 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:19:35 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:19:35 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:19:35 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:19:35 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:19:35 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:19:35 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:19:35 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:19:40 oak-gw06 kernel: warn_alloc_failed: 1548 callbacks suppressed Aug 1 10:19:40 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:40 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:40 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:40 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:40 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:40 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8801c4d8bc00 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa0000 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: Call Trace: Aug 1 10:19:40 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:40 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:40 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:40 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:40 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:40 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:40 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:40 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:40 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:40 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:19:40 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:19:40 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:19:40 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:19:40 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:40 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:40 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:40 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:40 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8801c4d8bc00 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa0000 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: Call Trace: Aug 1 10:19:40 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:40 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:40 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:40 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:40 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:40 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:40 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:40 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:40 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:40 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:19:40 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:19:40 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:19:40 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:19:40 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:40 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:40 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:40 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:40 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8801c4d8bc00 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa0000 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: Call Trace: Aug 1 10:19:40 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:40 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:40 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:40 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:40 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:40 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:40 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:40 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:40 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:40 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:19:40 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:19:40 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:19:40 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:19:40 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:40 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:40 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:40 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:40 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8801c4d8bc00 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa0000 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: Call Trace: Aug 1 10:19:40 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:40 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:40 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:40 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:40 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:40 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:40 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:40 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:40 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:40 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:19:40 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:19:40 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:19:40 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:19:40 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:40 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:40 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:40 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:40 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8801c4d8bc00 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa0000 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: Call Trace: Aug 1 10:19:40 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:40 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:40 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:40 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:40 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:40 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:40 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:40 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:40 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:40 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:19:40 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:19:40 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:19:40 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:19:40 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:40 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:40 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:40 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:40 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8801c4d8bc00 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa0000 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: Call Trace: Aug 1 10:19:40 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:40 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:40 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:40 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:40 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:40 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:40 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:40 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:40 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:40 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:19:40 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:19:40 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:19:40 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:19:40 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:40 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:40 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:40 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:40 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8801c4d8bc00 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa0000 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: Call Trace: Aug 1 10:19:40 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:40 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:40 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:40 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:40 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:40 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:40 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:40 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:40 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:40 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:19:40 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:19:40 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:19:40 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:19:40 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:40 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:40 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:40 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:40 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8801c4d8bc00 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa0000 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: Call Trace: Aug 1 10:19:40 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:40 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:40 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:40 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:40 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:40 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:40 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:40 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:40 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:40 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:19:40 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:19:40 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:19:40 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:19:40 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:19:40 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:40 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:40 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:19:40 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8801c4d8bc00 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa0000 a5232eaa14f210cb Aug 1 10:19:40 oak-gw06 kernel: Call Trace: Aug 1 10:19:40 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:40 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:40 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:40 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:40 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:40 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:40 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:40 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:40 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:40 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:40 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:40 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:40 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:19:40 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:19:40 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:19:40 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:19:40 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:19:41 oak-gw06 kernel: CPU: 4 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:19:41 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:19:41 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fd039d8 ffffffff8168662f Aug 1 10:19:41 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:19:41 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4edd00 00000000210cf37f Aug 1 10:19:41 oak-gw06 kernel: Call Trace: Aug 1 10:19:41 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:19:41 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:19:41 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:19:41 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:19:41 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:19:41 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:19:41 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:19:41 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:19:41 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:19:41 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:19:41 oak-gw06 kernel: [] ? bnx2x_rx_int+0xe0d/0x17b0 [bnx2x] Aug 1 10:19:41 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:19:41 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:19:41 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:19:41 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:19:41 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:19:41 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:19:41 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:19:41 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:19:41 oak-gw06 kernel: [] ? __list_del_entry+0x29/0xd0 Aug 1 10:19:41 oak-gw06 kernel: [] __osc_lru_del+0x2f/0x80 [osc] Aug 1 10:19:41 oak-gw06 kernel: [] osc_page_delete+0x115/0x4e0 [osc] Aug 1 10:19:41 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:19:41 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:19:41 oak-gw06 kernel: [] ll_releasepage+0xee/0x1a0 [lustre] Aug 1 10:19:41 oak-gw06 kernel: [] try_to_release_page+0x32/0x50 Aug 1 10:19:41 oak-gw06 kernel: [] shrink_page_list+0x950/0xb00 Aug 1 10:19:41 oak-gw06 kernel: [] shrink_inactive_list+0x1fa/0x630 Aug 1 10:19:41 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:19:41 oak-gw06 kernel: [] ? ldlm_cli_pool_shrink+0x72/0x100 [ptlrpc] Aug 1 10:19:41 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:19:41 oak-gw06 kernel: [] balance_pgdat+0x48c/0x5e0 Aug 1 10:19:41 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:19:41 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:19:41 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:19:41 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:19:41 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:19:41 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:19:41 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:13 oak-gw06 kernel: warn_alloc_failed: 249 callbacks suppressed Aug 1 10:20:13 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 10:20:13 oak-gw06 kernel: CPU: 7 PID: 11865 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:13 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:13 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:20:13 oak-gw06 kernel: 00000000000080d0 000000003a207b03 ffff88026f1e7858 ffffffff8168662f Aug 1 10:20:13 oak-gw06 kernel: ffff88026f1e78e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 10:20:13 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88026f1e78b8 000000003a207b03 Aug 1 10:20:13 oak-gw06 kernel: Call Trace: Aug 1 10:20:13 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:13 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:13 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 10:20:13 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 10:20:13 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:20:13 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:20:13 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:20:13 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:20:13 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:20:13 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:20:13 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:20:13 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:20:13 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:20:13 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:20:13 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:20:13 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:20:13 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:13 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:13 oak-gw06 kernel: Mem-Info: Aug 1 10:20:13 oak-gw06 kernel: active_anon:28175 inactive_anon:41519 isolated_anon:0#012 active_file:788187 inactive_file:1868466 isolated_file:0#012 unevictable:0 dirty:15972 writeback:5128 unstable:0#012 slab_reclaimable:42337 slab_unreclaimable:914817#012 mapped:5314 shmem:39000 pagetables:1651 bounce:0#012 free:229081 free_pcp:236 free_cma:0 Aug 1 10:20:13 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:20:13 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:20:13 oak-gw06 kernel: Node 0 DMA32 free:216368kB min:11976kB low:14968kB high:17964kB active_anon:17796kB inactive_anon:31248kB active_file:547704kB inactive_file:1299564kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:11608kB writeback:4368kB mapped:1996kB shmem:31276kB slab_reclaimable:30580kB slab_unreclaimable:644396kB kernel_stack:1024kB pagetables:1064kB unstable:0kB bounce:0kB free_pcp:300kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:20:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:20:13 oak-gw06 kernel: Node 0 Normal free:677356kB min:55536kB low:69420kB high:83304kB active_anon:94904kB inactive_anon:134828kB active_file:2608684kB inactive_file:6171700kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:51892kB writeback:15368kB mapped:19260kB shmem:124724kB slab_reclaimable:138768kB slab_unreclaimable:3014856kB kernel_stack:4672kB pagetables:5540kB unstable:0kB bounce:0kB free_pcp:1236kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:20:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:20:13 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:20:13 oak-gw06 kernel: Node 0 DMA32: 151*4kB (UE) 1488*8kB (UEM) 4842*16kB (UEM) 3513*32kB (UEM) 162*64kB (UEM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 212892kB Aug 1 10:20:13 oak-gw06 kernel: Node 0 Normal: 765*4kB (EM) 3759*8kB (UEM) 35686*16kB (UEM) 1858*32kB (UEM) 27*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 665292kB Aug 1 10:20:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:20:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:20:13 oak-gw06 kernel: 2081609 total pagecache pages Aug 1 10:20:13 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:20:13 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:20:13 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:20:13 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:20:13 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:20:13 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:20:13 oak-gw06 kernel: 127313 pages reserved Aug 1 10:20:13 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 10:20:13 oak-gw06 kernel: CPU: 7 PID: 11865 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:13 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:13 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:20:13 oak-gw06 kernel: 00000000000080d0 000000003a207b03 ffff88026f1e7808 ffffffff8168662f Aug 1 10:20:13 oak-gw06 kernel: ffff88026f1e7898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 10:20:13 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88026f1e7868 000000003a207b03 Aug 1 10:20:13 oak-gw06 kernel: Call Trace: Aug 1 10:20:13 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:13 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:13 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:13 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:13 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 10:20:13 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 10:20:13 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:20:13 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:20:13 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:20:13 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:20:13 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:20:13 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:20:13 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:20:13 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:20:13 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:20:13 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:20:13 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:20:13 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:20:13 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:13 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:13 oak-gw06 kernel: Mem-Info: Aug 1 10:20:13 oak-gw06 kernel: active_anon:28175 inactive_anon:41519 isolated_anon:0#012 active_file:793222 inactive_file:1867512 isolated_file:0#012 unevictable:0 dirty:15875 writeback:4934 unstable:0#012 slab_reclaimable:42337 slab_unreclaimable:914817#012 mapped:5314 shmem:39000 pagetables:1651 bounce:0#012 free:218769 free_pcp:244 free_cma:0 Aug 1 10:20:13 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:20:13 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:20:13 oak-gw06 kernel: Node 0 DMA32 free:208324kB min:11976kB low:14968kB high:17964kB active_anon:17796kB inactive_anon:31248kB active_file:551464kB inactive_file:1297308kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:11608kB writeback:4368kB mapped:1996kB shmem:31276kB slab_reclaimable:30580kB slab_unreclaimable:644396kB kernel_stack:1024kB pagetables:1064kB unstable:0kB bounce:0kB free_pcp:408kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:20:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:20:13 oak-gw06 kernel: Node 0 Normal free:634264kB min:55536kB low:69420kB high:83304kB active_anon:94904kB inactive_anon:134828kB active_file:2625844kB inactive_file:6177940kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:51892kB writeback:16144kB mapped:19260kB shmem:124724kB slab_reclaimable:138768kB slab_unreclaimable:3014856kB kernel_stack:4672kB pagetables:5540kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:20:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:20:13 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:20:13 oak-gw06 kernel: Node 0 DMA32: 150*4kB (E) 1059*8kB (UEM) 4603*16kB (UEM) 3513*32kB (UEM) 162*64kB (UEM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 205632kB Aug 1 10:20:13 oak-gw06 kernel: Node 0 Normal: 765*4kB (EM) 1485*8kB (UE) 34574*16kB (UEM) 1858*32kB (UEM) 27*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 629308kB Aug 1 10:20:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:20:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:20:13 oak-gw06 kernel: 2086599 total pagecache pages Aug 1 10:20:13 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:20:13 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:20:13 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:20:13 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:20:13 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:20:13 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:20:13 oak-gw06 kernel: 127313 pages reserved Aug 1 10:20:22 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:22 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:22 oak-gw06 kernel: CPU: 4 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:22 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fd039d8 ffffffff8168662f Aug 1 10:20:22 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:20:22 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff880426f92000 00000000da86253b Aug 1 10:20:22 oak-gw06 kernel: Call Trace: Aug 1 10:20:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:20:22 oak-gw06 kernel: [] ? __dev_kfree_skb_any+0x3d/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] ? bnx2x_tx_int+0xc8/0x220 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:22 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:22 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:22 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:22 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:22 oak-gw06 kernel: [] ? loop_64+0x27/0x78 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:20:22 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:20:22 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? lustre_swab_obd_ioobj+0x30/0x30 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_brw_prep_request+0xa61/0xf10 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_build_rpc+0x555/0xfb0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_check_rpcs+0xe2b/0x18b0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? default_wake_function+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? __wake_up_common+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] ? update_rq_clock.part.78+0x4c/0x150 Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug0+0xe2/0x130 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug+0x10/0x20 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] brw_queue_work+0x31/0xd0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:22 oak-gw06 kernel: CPU: 4 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:22 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fd039d8 ffffffff8168662f Aug 1 10:20:22 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:20:22 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff880426f92000 00000000da86253b Aug 1 10:20:22 oak-gw06 kernel: Call Trace: Aug 1 10:20:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:20:22 oak-gw06 kernel: [] ? bnx2x_tx_int+0xc8/0x220 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:22 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:22 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:22 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:22 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:22 oak-gw06 kernel: [] ? loop_64+0x27/0x78 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:20:22 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:20:22 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? lustre_swab_obd_ioobj+0x30/0x30 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_brw_prep_request+0xa61/0xf10 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_build_rpc+0x555/0xfb0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_check_rpcs+0xe2b/0x18b0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? default_wake_function+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? __wake_up_common+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] ? update_rq_clock.part.78+0x4c/0x150 Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug0+0xe2/0x130 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug+0x10/0x20 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] brw_queue_work+0x31/0xd0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:22 oak-gw06 kernel: CPU: 4 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:22 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fd039d8 ffffffff8168662f Aug 1 10:20:22 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:20:22 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff880426f92000 00000000da86253b Aug 1 10:20:22 oak-gw06 kernel: Call Trace: Aug 1 10:20:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:22 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:22 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:22 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:22 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:22 oak-gw06 kernel: [] ? loop_64+0x27/0x78 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:20:22 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:20:22 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? lustre_swab_obd_ioobj+0x30/0x30 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_brw_prep_request+0xa61/0xf10 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_build_rpc+0x555/0xfb0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_check_rpcs+0xe2b/0x18b0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? default_wake_function+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? __wake_up_common+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] ? update_rq_clock.part.78+0x4c/0x150 Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug0+0xe2/0x130 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug+0x10/0x20 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] brw_queue_work+0x31/0xd0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:22 oak-gw06 kernel: CPU: 4 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:22 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fd039d8 ffffffff8168662f Aug 1 10:20:22 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:20:22 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff880426f92000 00000000da86253b Aug 1 10:20:22 oak-gw06 kernel: Call Trace: Aug 1 10:20:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:22 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:22 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:22 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:22 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:22 oak-gw06 kernel: [] ? loop_64+0x27/0x78 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:20:22 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:20:22 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? lustre_swab_obd_ioobj+0x30/0x30 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_brw_prep_request+0xa61/0xf10 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_build_rpc+0x555/0xfb0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_check_rpcs+0xe2b/0x18b0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? default_wake_function+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? __wake_up_common+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] ? update_rq_clock.part.78+0x4c/0x150 Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug0+0xe2/0x130 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug+0x10/0x20 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] brw_queue_work+0x31/0xd0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:22 oak-gw06 kernel: CPU: 4 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:22 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fd039d8 ffffffff8168662f Aug 1 10:20:22 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:20:22 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff880426f92000 00000000da86253b Aug 1 10:20:22 oak-gw06 kernel: Call Trace: Aug 1 10:20:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:22 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:22 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:22 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:22 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:22 oak-gw06 kernel: [] ? loop_64+0x27/0x78 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:20:22 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:20:22 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? lustre_swab_obd_ioobj+0x30/0x30 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_brw_prep_request+0xa61/0xf10 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_build_rpc+0x555/0xfb0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_check_rpcs+0xe2b/0x18b0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? default_wake_function+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? __wake_up_common+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] ? update_rq_clock.part.78+0x4c/0x150 Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug0+0xe2/0x130 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug+0x10/0x20 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] brw_queue_work+0x31/0xd0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:22 oak-gw06 kernel: CPU: 4 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:22 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fd039d8 ffffffff8168662f Aug 1 10:20:22 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88043fd03a80 ffffffff815d720c Aug 1 10:20:22 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffffffff81aebbd0 00000000da86253b Aug 1 10:20:22 oak-gw06 kernel: Call Trace: Aug 1 10:20:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:22 oak-gw06 kernel: [] ? tcp_v4_rcv+0x7ac/0x9a0 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:22 oak-gw06 kernel: [] ? ip_local_deliver_finish+0xb4/0x1f0 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:22 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:22 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:22 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:22 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:22 oak-gw06 kernel: [] ? loop_64+0x27/0x78 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:20:22 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:20:22 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? lustre_swab_obd_ioobj+0x30/0x30 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_brw_prep_request+0xa61/0xf10 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_build_rpc+0x555/0xfb0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_check_rpcs+0xe2b/0x18b0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? default_wake_function+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? __wake_up_common+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] ? update_rq_clock.part.78+0x4c/0x150 Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug0+0xe2/0x130 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug+0x10/0x20 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] brw_queue_work+0x31/0xd0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:22 oak-gw06 kernel: CPU: 4 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:22 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fd039d8 ffffffff8168662f Aug 1 10:20:22 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88043fd03a80 ffffffff815d720c Aug 1 10:20:22 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffffffff81aebbd0 00000000da86253b Aug 1 10:20:22 oak-gw06 kernel: Call Trace: Aug 1 10:20:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:22 oak-gw06 kernel: [] ? tcp_v4_rcv+0x7ac/0x9a0 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:22 oak-gw06 kernel: [] ? ip_local_deliver_finish+0xb4/0x1f0 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:22 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:22 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:22 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:22 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:22 oak-gw06 kernel: [] ? loop_64+0x27/0x78 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:20:22 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:20:22 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? lustre_swab_obd_ioobj+0x30/0x30 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_brw_prep_request+0xa61/0xf10 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_build_rpc+0x555/0xfb0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_check_rpcs+0xe2b/0x18b0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? default_wake_function+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? __wake_up_common+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] ? update_rq_clock.part.78+0x4c/0x150 Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug0+0xe2/0x130 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug+0x10/0x20 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] brw_queue_work+0x31/0xd0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:22 oak-gw06 kernel: CPU: 4 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:22 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fd039d8 ffffffff8168662f Aug 1 10:20:22 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88043fd03a80 ffffffff815d720c Aug 1 10:20:22 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffffffff81aebbd0 00000000da86253b Aug 1 10:20:22 oak-gw06 kernel: Call Trace: Aug 1 10:20:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:22 oak-gw06 kernel: [] ? tcp_v4_rcv+0x7ac/0x9a0 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:22 oak-gw06 kernel: [] ? ip_local_deliver_finish+0xb4/0x1f0 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:22 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:22 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:22 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:22 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:22 oak-gw06 kernel: [] ? loop_64+0x27/0x78 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:20:22 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:20:22 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? lustre_swab_obd_ioobj+0x30/0x30 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_brw_prep_request+0xa61/0xf10 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_build_rpc+0x555/0xfb0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_check_rpcs+0xe2b/0x18b0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? default_wake_function+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? __wake_up_common+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] ? update_rq_clock.part.78+0x4c/0x150 Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug0+0xe2/0x130 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug+0x10/0x20 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] brw_queue_work+0x31/0xd0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:22 oak-gw06 kernel: CPU: 4 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:22 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fd039d8 ffffffff8168662f Aug 1 10:20:22 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88043fd03a80 ffffffff815d720c Aug 1 10:20:22 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffffffff81aebbd0 00000000da86253b Aug 1 10:20:22 oak-gw06 kernel: Call Trace: Aug 1 10:20:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:22 oak-gw06 kernel: [] ? tcp_v4_rcv+0x7ac/0x9a0 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:22 oak-gw06 kernel: [] ? ip_local_deliver_finish+0xb4/0x1f0 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:22 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:22 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:22 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:22 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:22 oak-gw06 kernel: [] ? loop_64+0x27/0x78 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:20:22 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:20:22 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:20:22 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? lustre_swab_obd_ioobj+0x30/0x30 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_brw_prep_request+0xa61/0xf10 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_build_rpc+0x555/0xfb0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_check_rpcs+0xe2b/0x18b0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] ? default_wake_function+0x12/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? __wake_up_common+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? sched_clock_cpu+0x85/0xc0 Aug 1 10:20:22 oak-gw06 kernel: [] ? update_rq_clock.part.78+0x4c/0x150 Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug0+0xe2/0x130 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] osc_io_unplug+0x10/0x20 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] brw_queue_work+0x31/0xd0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:20:22 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:20:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:22 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:20:22 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:20:22 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa45c0 00000000210cf37f Aug 1 10:20:22 oak-gw06 kernel: Call Trace: Aug 1 10:20:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:20:22 oak-gw06 kernel: [] ? __dev_kfree_skb_any+0x3d/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:22 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:22 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:22 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:22 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:22 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:22 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:22 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:22 oak-gw06 kernel: [] ? __free_memcg_kmem_pages+0x22/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] ? _raw_spin_lock+0x32/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] osc_page_delete+0xf6/0x4e0 [osc] Aug 1 10:20:22 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:20:22 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:20:22 oak-gw06 kernel: [] ll_releasepage+0xee/0x1a0 [lustre] Aug 1 10:20:22 oak-gw06 kernel: [] try_to_release_page+0x32/0x50 Aug 1 10:20:22 oak-gw06 kernel: [] shrink_page_list+0x950/0xb00 Aug 1 10:20:22 oak-gw06 kernel: [] shrink_inactive_list+0x1fa/0x630 Aug 1 10:20:22 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:20:22 oak-gw06 kernel: [] ? cl_env_put+0x140/0x1d0 [obdclass] Aug 1 10:20:22 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:20:22 oak-gw06 kernel: [] balance_pgdat+0x48c/0x5e0 Aug 1 10:20:22 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:20:22 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:20:22 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:20:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: warn_alloc_failed: 1306 callbacks suppressed Aug 1 10:20:37 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:37 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:37 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:37 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:20:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffffa05fb54d 0000000000000002 Aug 1 10:20:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88017a28c000 00000000210cf37f Aug 1 10:20:37 oak-gw06 kernel: Call Trace: Aug 1 10:20:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:37 oak-gw06 kernel: [] ? tcp_in_window+0xfd/0xa60 [nf_conntrack] Aug 1 10:20:37 oak-gw06 kernel: [] ? internal_add_timer+0x32/0x70 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] ? ip_queue_xmit+0x143/0x3a0 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:37 oak-gw06 kernel: [] ? memset+0x6/0xb0 Aug 1 10:20:37 oak-gw06 kernel: [] ? lov_fini_raid0+0x11c/0x2d0 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lov_object_free+0x79/0x470 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_free.isra.31+0x11f/0x1a0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_put+0xc2/0x3d0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_object_put+0xe/0x10 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_inode_fini+0x75/0x220 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_clear_inode+0x20c/0x820 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_delete_inode+0x58/0x1c0 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:20:37 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:20:37 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:20:37 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:20:37 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:37 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:37 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:20:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffffa05fb54d 0000000000000002 Aug 1 10:20:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88017a28c000 00000000210cf37f Aug 1 10:20:37 oak-gw06 kernel: Call Trace: Aug 1 10:20:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:37 oak-gw06 kernel: [] ? tcp_in_window+0xfd/0xa60 [nf_conntrack] Aug 1 10:20:37 oak-gw06 kernel: [] ? internal_add_timer+0x32/0x70 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] ? ip_queue_xmit+0x143/0x3a0 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:37 oak-gw06 kernel: [] ? memset+0x6/0xb0 Aug 1 10:20:37 oak-gw06 kernel: [] ? lov_fini_raid0+0x11c/0x2d0 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lov_object_free+0x79/0x470 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_free.isra.31+0x11f/0x1a0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_put+0xc2/0x3d0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_object_put+0xe/0x10 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_inode_fini+0x75/0x220 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_clear_inode+0x20c/0x820 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_delete_inode+0x58/0x1c0 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:20:37 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:20:37 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:20:37 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:20:37 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:37 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:37 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:20:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffffa05fb54d 0000000000000002 Aug 1 10:20:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88017a28c000 00000000210cf37f Aug 1 10:20:37 oak-gw06 kernel: Call Trace: Aug 1 10:20:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:37 oak-gw06 kernel: [] ? tcp_in_window+0xfd/0xa60 [nf_conntrack] Aug 1 10:20:37 oak-gw06 kernel: [] ? internal_add_timer+0x32/0x70 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] ? ip_queue_xmit+0x143/0x3a0 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:37 oak-gw06 kernel: [] ? memset+0x6/0xb0 Aug 1 10:20:37 oak-gw06 kernel: [] ? lov_fini_raid0+0x11c/0x2d0 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lov_object_free+0x79/0x470 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_free.isra.31+0x11f/0x1a0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_put+0xc2/0x3d0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_object_put+0xe/0x10 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_inode_fini+0x75/0x220 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_clear_inode+0x20c/0x820 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_delete_inode+0x58/0x1c0 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:20:37 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:20:37 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:20:37 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:20:37 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:37 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:37 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:20:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffffa05fb54d 0000000000000002 Aug 1 10:20:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88017a28c000 00000000210cf37f Aug 1 10:20:37 oak-gw06 kernel: Call Trace: Aug 1 10:20:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:37 oak-gw06 kernel: [] ? tcp_in_window+0xfd/0xa60 [nf_conntrack] Aug 1 10:20:37 oak-gw06 kernel: [] ? internal_add_timer+0x32/0x70 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] ? ip_queue_xmit+0x143/0x3a0 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:37 oak-gw06 kernel: [] ? memset+0x6/0xb0 Aug 1 10:20:37 oak-gw06 kernel: [] ? lov_fini_raid0+0x11c/0x2d0 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lov_object_free+0x79/0x470 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_free.isra.31+0x11f/0x1a0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_put+0xc2/0x3d0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_object_put+0xe/0x10 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_inode_fini+0x75/0x220 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_clear_inode+0x20c/0x820 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_delete_inode+0x58/0x1c0 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:20:37 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:20:37 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:20:37 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:20:37 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:37 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:37 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:20:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffffa05fb54d 0000000000000002 Aug 1 10:20:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88017a28c000 00000000210cf37f Aug 1 10:20:37 oak-gw06 kernel: Call Trace: Aug 1 10:20:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:37 oak-gw06 kernel: [] ? tcp_in_window+0xfd/0xa60 [nf_conntrack] Aug 1 10:20:37 oak-gw06 kernel: [] ? internal_add_timer+0x32/0x70 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] ? ip_queue_xmit+0x143/0x3a0 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:37 oak-gw06 kernel: [] ? memset+0x6/0xb0 Aug 1 10:20:37 oak-gw06 kernel: [] ? lov_fini_raid0+0x11c/0x2d0 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lov_object_free+0x79/0x470 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_free.isra.31+0x11f/0x1a0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_put+0xc2/0x3d0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_object_put+0xe/0x10 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_inode_fini+0x75/0x220 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_clear_inode+0x20c/0x820 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_delete_inode+0x58/0x1c0 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:20:37 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:20:37 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:20:37 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:20:37 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:37 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:37 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:20:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffffa05fb54d 0000000000000002 Aug 1 10:20:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88017a28c000 00000000210cf37f Aug 1 10:20:37 oak-gw06 kernel: Call Trace: Aug 1 10:20:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:37 oak-gw06 kernel: [] ? tcp_in_window+0xfd/0xa60 [nf_conntrack] Aug 1 10:20:37 oak-gw06 kernel: [] ? internal_add_timer+0x32/0x70 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] ? ip_queue_xmit+0x143/0x3a0 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:37 oak-gw06 kernel: [] ? memset+0x6/0xb0 Aug 1 10:20:37 oak-gw06 kernel: [] ? lov_fini_raid0+0x11c/0x2d0 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lov_object_free+0x79/0x470 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_free.isra.31+0x11f/0x1a0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_put+0xc2/0x3d0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_object_put+0xe/0x10 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_inode_fini+0x75/0x220 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_clear_inode+0x20c/0x820 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_delete_inode+0x58/0x1c0 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:20:37 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:20:37 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:20:37 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:20:37 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:37 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:37 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:20:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffffa05fb54d 0000000000000002 Aug 1 10:20:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88017a28c000 00000000210cf37f Aug 1 10:20:37 oak-gw06 kernel: Call Trace: Aug 1 10:20:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:37 oak-gw06 kernel: [] ? tcp_in_window+0xfd/0xa60 [nf_conntrack] Aug 1 10:20:37 oak-gw06 kernel: [] ? internal_add_timer+0x32/0x70 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] ? ip_queue_xmit+0x143/0x3a0 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:37 oak-gw06 kernel: [] ? memset+0x6/0xb0 Aug 1 10:20:37 oak-gw06 kernel: [] ? lov_fini_raid0+0x11c/0x2d0 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lov_object_free+0x79/0x470 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_free.isra.31+0x11f/0x1a0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_put+0xc2/0x3d0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_object_put+0xe/0x10 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_inode_fini+0x75/0x220 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_clear_inode+0x20c/0x820 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_delete_inode+0x58/0x1c0 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:20:37 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:20:37 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:20:37 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:20:37 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:37 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:37 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:20:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffffa05fb54d 0000000000000002 Aug 1 10:20:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88017a28c000 00000000210cf37f Aug 1 10:20:37 oak-gw06 kernel: Call Trace: Aug 1 10:20:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:37 oak-gw06 kernel: [] ? tcp_in_window+0xfd/0xa60 [nf_conntrack] Aug 1 10:20:37 oak-gw06 kernel: [] ? internal_add_timer+0x32/0x70 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] ? ip_queue_xmit+0x143/0x3a0 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:37 oak-gw06 kernel: [] ? memset+0x6/0xb0 Aug 1 10:20:37 oak-gw06 kernel: [] ? lov_fini_raid0+0x11c/0x2d0 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lov_object_free+0x79/0x470 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_free.isra.31+0x11f/0x1a0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_put+0xc2/0x3d0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_object_put+0xe/0x10 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_inode_fini+0x75/0x220 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_clear_inode+0x20c/0x820 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_delete_inode+0x58/0x1c0 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:20:37 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:20:37 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:20:37 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:20:37 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:20:37 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:37 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:20:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffffa05fb54d 0000000000000002 Aug 1 10:20:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88017a28c000 00000000210cf37f Aug 1 10:20:37 oak-gw06 kernel: Call Trace: Aug 1 10:20:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:37 oak-gw06 kernel: [] ? tcp_in_window+0xfd/0xa60 [nf_conntrack] Aug 1 10:20:37 oak-gw06 kernel: [] ? internal_add_timer+0x32/0x70 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] ? ip_queue_xmit+0x143/0x3a0 Aug 1 10:20:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:37 oak-gw06 kernel: [] ? memset+0x6/0xb0 Aug 1 10:20:37 oak-gw06 kernel: [] ? lov_fini_raid0+0x11c/0x2d0 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lov_object_free+0x79/0x470 [lov] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_free.isra.31+0x11f/0x1a0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] lu_object_put+0xc2/0x3d0 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_object_put+0xe/0x10 [obdclass] Aug 1 10:20:37 oak-gw06 kernel: [] cl_inode_fini+0x75/0x220 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_clear_inode+0x20c/0x820 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ll_delete_inode+0x58/0x1c0 [lustre] Aug 1 10:20:37 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:20:37 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:20:37 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:20:37 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:20:37 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:20:37 oak-gw06 kernel: [] ? vmpressure+0x87/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:20:37 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:20:37 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:20:37 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:37 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:20:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:20:38 oak-gw06 kernel: CPU: 4 PID: 11863 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:20:38 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:20:38 oak-gw06 kernel: 0000000000104020 0000000003744db4 ffff88043fd039d8 ffffffff8168662f Aug 1 10:20:38 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:20:38 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9f00 0000000003744db4 Aug 1 10:20:38 oak-gw06 kernel: Call Trace: Aug 1 10:20:38 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:20:38 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:20:38 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:20:38 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:20:38 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:20:38 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:20:38 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:20:38 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:20:38 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:20:38 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:20:38 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:20:38 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:20:38 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:20:38 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:20:38 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:20:38 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:20:38 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:20:38 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:20:38 oak-gw06 kernel: [] ? sysret_audit+0x17/0x21 Aug 1 10:21:04 oak-gw06 kernel: warn_alloc_failed: 1288 callbacks suppressed Aug 1 10:21:04 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:21:04 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:21:04 oak-gw06 kernel: CPU: 4 PID: 11863 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:21:04 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:21:04 oak-gw06 kernel: 0000000000104020 0000000003744db4 ffff88043fd039d8 ffffffff8168662f Aug 1 10:21:04 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:21:04 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9f00 0000000003744db4 Aug 1 10:21:04 oak-gw06 kernel: Call Trace: Aug 1 10:21:04 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:21:04 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:21:04 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:21:04 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:21:04 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:21:04 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:21:04 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:21:04 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:21:04 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:21:04 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:21:04 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:21:04 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:21:04 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:21:04 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:21:04 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:21:04 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:21:04 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:21:04 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:21:04 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:21:04 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:21:04 oak-gw06 kernel: Aug 1 10:21:04 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:21:04 oak-gw06 kernel: CPU: 4 PID: 11863 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:21:04 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:21:04 oak-gw06 kernel: 0000000000104020 0000000003744db4 ffff88043fd039d8 ffffffff8168662f Aug 1 10:21:04 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:21:04 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9f00 0000000003744db4 Aug 1 10:21:04 oak-gw06 kernel: Call Trace: Aug 1 10:21:04 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:21:04 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:21:04 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:21:04 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:21:04 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:21:04 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:21:04 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:21:04 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:21:04 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:21:04 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:21:04 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:21:04 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:21:04 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:21:04 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:21:04 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:21:04 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:21:04 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:21:04 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:21:04 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:21:04 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:21:04 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:21:04 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:21:04 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:21:04 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:21:04 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:21:04 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:21:04 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:21:04 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:21:04 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:21:04 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:21:04 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:21:04 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:21:04 oak-gw06 kernel: Aug 1 10:21:04 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:21:04 oak-gw06 kernel: CPU: 4 PID: 11863 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:21:04 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:21:04 oak-gw06 kernel: 0000000000104020 0000000003744db4 ffff88043fd039d8 ffffffff8168662f Aug 1 10:21:04 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:21:04 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9f00 0000000003744db4 Aug 1 10:21:04 oak-gw06 kernel: Call Trace: Aug 1 10:21:04 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:21:04 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:21:04 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:21:04 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:21:04 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:21:04 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:21:04 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:21:04 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:21:04 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:21:04 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:21:04 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:21:04 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:21:04 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:21:04 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:21:04 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:21:04 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:21:04 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:21:04 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:21:04 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:21:04 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:21:04 oak-gw06 kernel: Aug 1 10:21:04 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:21:04 oak-gw06 kernel: CPU: 4 PID: 11863 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:21:04 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:21:04 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:21:04 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:21:04 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:21:04 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:21:04 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:21:04 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:21:04 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:21:04 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:21:04 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:21:05 oak-gw06 kernel: CPU: 3 PID: 11860 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:21:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:21:05 oak-gw06 kernel: Aug 1 10:21:05 oak-gw06 kernel: 0000000000104020 000000008f78d2e5 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:21:05 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8800a2798050 ffff880415b500b8 Aug 1 10:21:05 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa07c0 000000008f78d2e5 Aug 1 10:21:05 oak-gw06 kernel: Call Trace: Aug 1 10:21:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:21:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:21:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:21:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:21:05 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:21:05 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:21:05 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:21:05 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:21:05 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:21:05 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:21:05 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:21:05 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:21:05 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:21:05 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:21:05 oak-gw06 kernel: [] ? ll_merge_attr+0x23/0x390 [lustre] Aug 1 10:21:05 oak-gw06 kernel: [] ? cl_io_commit_async+0x77/0x140 [obdclass] Aug 1 10:21:05 oak-gw06 kernel: [] vvp_io_write_commit+0x3f6/0x8d0 [lustre] Aug 1 10:21:05 oak-gw06 kernel: [] vvp_io_write_start+0x54a/0x720 [lustre] Aug 1 10:21:05 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:21:05 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:21:05 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:21:05 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:21:05 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:21:05 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:21:05 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:21:05 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:21:05 oak-gw06 kernel: swapper/4: page allocation failure: order:2, mode:0x104020 Aug 1 10:21:05 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:21:05 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:21:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:21:05 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:21:05 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:21:05 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa0000 a5232eaa14f210cb Aug 1 10:21:05 oak-gw06 kernel: Call Trace: Aug 1 10:21:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:21:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:21:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:21:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:21:05 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:21:05 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:21:05 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:21:05 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:21:05 oak-gw06 kernel: [] ? __dev_kfree_skb_any+0x3d/0x50 Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:21:05 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:21:05 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:21:05 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:21:05 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:21:05 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:21:05 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:21:05 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:21:05 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:21:05 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:21:05 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:21:05 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:21:05 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:21:05 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:21:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:21:05 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:21:05 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:21:05 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa0000 a5232eaa14f210cb Aug 1 10:21:05 oak-gw06 kernel: Call Trace: Aug 1 10:21:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:21:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:21:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:21:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:21:05 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:21:05 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:21:05 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:21:05 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:21:05 oak-gw06 kernel: [] ? __dev_kfree_skb_any+0x3d/0x50 Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:21:05 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:21:05 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:21:05 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:21:05 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:21:05 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:21:05 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:21:05 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:21:05 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:21:05 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:21:05 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:21:05 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:21:05 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:21:05 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:21:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:21:05 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:21:05 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:21:05 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa0000 a5232eaa14f210cb Aug 1 10:21:05 oak-gw06 kernel: Call Trace: Aug 1 10:21:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:21:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:21:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:21:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:21:05 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:21:05 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:21:05 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:21:05 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:21:05 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:21:05 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:21:05 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:21:05 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:21:05 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:21:05 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:21:05 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:21:05 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:21:05 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:21:05 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:21:05 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:21:05 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:21:05 oak-gw06 kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:21:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:21:05 oak-gw06 kernel: 0000000000104020 803b52a30d0a83c1 ffff88043fd039d8 ffffffff8168662f Aug 1 10:21:05 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff8800b15a4050 ffff880415b500b8 Aug 1 10:21:05 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4eae80 803b52a30d0a83c1 Aug 1 10:21:05 oak-gw06 kernel: Call Trace: Aug 1 10:21:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:21:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:21:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:21:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:21:05 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:21:05 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:21:05 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:21:05 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] ? napi_gro_complete+0x7d/0x100 Aug 1 10:21:05 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:21:05 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:21:05 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:21:05 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:21:05 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:21:05 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:21:05 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:21:05 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:21:05 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:21:05 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:21:05 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:21:05 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:21:05 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:22:07 oak-gw06 kernel: warn_alloc_failed: 255 callbacks suppressed Aug 1 10:22:07 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:22:07 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:22:07 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:22:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:22:07 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:22:07 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:22:07 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000282396d3 Aug 1 10:22:07 oak-gw06 kernel: Call Trace: Aug 1 10:22:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:22:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:22:07 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:22:07 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:22:07 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:22:07 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:22:07 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:22:07 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:22:07 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:22:07 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:22:07 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:22:07 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:22:07 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:22:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:22:07 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:22:07 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:22:07 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000282396d3 Aug 1 10:22:07 oak-gw06 kernel: Call Trace: Aug 1 10:22:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:22:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:22:07 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:22:07 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:22:07 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] ? timerqueue_add+0x60/0xb0 Aug 1 10:22:07 oak-gw06 kernel: [] ? kvm_clock_get_cycles+0x1f/0x30 Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:22:07 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:22:07 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:22:07 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:22:07 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:22:07 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:22:07 oak-gw06 kernel: Aug 1 10:22:07 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:22:07 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:22:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:22:07 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:22:07 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:22:07 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5540 00000000282396d3 Aug 1 10:22:07 oak-gw06 kernel: Call Trace: Aug 1 10:22:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:22:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:22:07 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:22:07 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:22:07 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:22:07 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] ? timerqueue_add+0x60/0xb0 Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:22:07 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:22:07 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:22:07 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:22:07 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:22:07 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:22:07 oak-gw06 kernel: Aug 1 10:22:07 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:22:07 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:22:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:22:07 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:22:07 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:22:07 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff880426f92000 00000000282396d3 Aug 1 10:22:07 oak-gw06 kernel: Call Trace: Aug 1 10:22:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:22:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:22:07 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:22:07 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:22:07 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] ? ttwu_do_wakeup+0x19/0xd0 Aug 1 10:22:07 oak-gw06 kernel: [] ? task_tick_fair+0x266/0x680 Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:22:07 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:22:07 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:22:07 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:22:07 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:22:07 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:22:07 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:22:07 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:22:07 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:22:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:22:07 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:22:07 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8802c5720050 ffff880415b500b8 Aug 1 10:22:07 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88043fcc3a18 00000000282396d3 Aug 1 10:22:07 oak-gw06 kernel: Call Trace: Aug 1 10:22:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:22:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:22:07 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:22:07 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:22:07 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:22:07 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:22:07 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:22:07 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:22:07 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:22:07 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:22:07 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:22:07 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:22:07 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:22:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:22:07 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:22:07 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8802c5720050 ffff880415b500b8 Aug 1 10:22:07 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88043fcc3a18 00000000282396d3 Aug 1 10:22:07 oak-gw06 kernel: Call Trace: Aug 1 10:22:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:22:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:22:07 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:22:07 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:22:07 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:22:07 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:22:07 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:22:07 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:22:07 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:22:07 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:22:07 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:22:07 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:22:07 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:22:07 oak-gw06 kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:22:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:22:07 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:22:07 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8802c5720050 ffff880415b500b8 Aug 1 10:22:07 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88043fcc3a18 00000000282396d3 Aug 1 10:22:07 oak-gw06 kernel: Call Trace: Aug 1 10:22:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:22:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:22:07 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:22:07 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:22:07 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:22:07 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:22:07 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:22:07 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:22:07 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:22:07 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:22:07 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:22:07 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:22:07 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:22:07 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:22:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:22:07 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:22:07 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8800387cc050 ffff880415b500b8 Aug 1 10:22:07 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa0000 a5232eaa14f210cb Aug 1 10:22:07 oak-gw06 kernel: Call Trace: Aug 1 10:22:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:22:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:22:07 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:22:07 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:22:07 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:22:07 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:22:07 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:22:07 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:22:07 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:22:07 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:22:07 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:22:07 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:22:07 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:22:07 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:22:07 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:22:07 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 1 10:22:07 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:22:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:22:07 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 1 10:22:07 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8800387cc050 ffff880415b500b8 Aug 1 10:22:07 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa0000 a5232eaa14f210cb Aug 1 10:22:07 oak-gw06 kernel: Call Trace: Aug 1 10:22:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:22:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:22:07 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:22:07 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:22:07 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:22:07 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:22:07 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:22:07 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:22:07 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:22:07 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:22:07 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:22:07 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:22:07 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:22:07 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:22:07 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:22:07 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:22:07 oak-gw06 kernel: CPU: 4 PID: 11863 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:22:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:22:07 oak-gw06 kernel: 0000000000104020 0000000003744db4 ffff88043fd039d8 ffffffff8168662f Aug 1 10:22:07 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 0000000000000010 0000000000000000 Aug 1 10:22:07 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4eec80 0000000003744db4 Aug 1 10:22:07 oak-gw06 kernel: Call Trace: Aug 1 10:22:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:22:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:22:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:22:07 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:22:07 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:22:07 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:22:07 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:22:07 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:22:07 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:22:07 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:22:07 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:22:07 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:22:07 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:22:07 oak-gw06 kernel: [] ? memcpy_toiovec+0x4a/0x90 Aug 1 10:22:07 oak-gw06 kernel: [] skb_copy_datagram_iovec+0x5b/0x2a0 Aug 1 10:22:07 oak-gw06 kernel: [] ? kmem_cache_free+0x1bb/0x1f0 Aug 1 10:22:07 oak-gw06 kernel: [] tcp_recvmsg+0x24a/0xb50 Aug 1 10:22:07 oak-gw06 kernel: [] ? osc_io_iter_fini+0x65/0xd0 [osc] Aug 1 10:22:07 oak-gw06 kernel: [] inet_recvmsg+0x7b/0xa0 Aug 1 10:22:07 oak-gw06 kernel: [] sock_recvmsg+0xbf/0x100 Aug 1 10:22:07 oak-gw06 kernel: [] SYSC_recvfrom+0xe8/0x160 Aug 1 10:22:07 oak-gw06 kernel: [] ? poll_select_copy_remaining+0xfc/0x150 Aug 1 10:22:07 oak-gw06 kernel: [] SyS_recvfrom+0xe/0x10 Aug 1 10:22:07 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:23:00 oak-gw06 kernel: warn_alloc_failed: 1053 callbacks suppressed Aug 1 10:23:00 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:23:00 oak-gw06 kernel: CPU: 4 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:23:00 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:23:00 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fd039d8 ffffffff8168662f Aug 1 10:23:00 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff880002388050 ffff880415b500b8 Aug 1 10:23:00 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9f00 00000000210cf37f Aug 1 10:23:00 oak-gw06 kernel: Call Trace: Aug 1 10:23:00 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:23:00 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:23:00 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:23:00 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:23:00 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:23:00 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:23:00 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:23:00 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:23:00 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:23:00 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:23:00 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:23:00 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:23:00 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:23:00 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:23:00 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:23:00 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:23:00 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:23:00 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:23:00 oak-gw06 kernel: [] ? cl_io_init0.isra.15+0x88/0x160 [obdclass] Aug 1 10:23:00 oak-gw06 kernel: [] ? cfs_hash_hd_hnode_del+0x3f/0x50 [libcfs] Aug 1 10:23:00 oak-gw06 kernel: [] cfs_hash_bd_del_locked+0x1e/0xf0 [libcfs] Aug 1 10:23:00 oak-gw06 kernel: [] lu_object_put+0x2de/0x3d0 [obdclass] Aug 1 10:23:00 oak-gw06 kernel: [] cl_object_put+0xe/0x10 [obdclass] Aug 1 10:23:00 oak-gw06 kernel: [] cl_inode_fini+0x75/0x220 [lustre] Aug 1 10:23:00 oak-gw06 kernel: [] ll_clear_inode+0x20c/0x820 [lustre] Aug 1 10:23:00 oak-gw06 kernel: [] ll_delete_inode+0x58/0x1c0 [lustre] Aug 1 10:23:00 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:23:00 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:23:00 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:23:00 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:23:00 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:23:00 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:23:00 oak-gw06 kernel: [] ? vmpressure+0x21/0x90 Aug 1 10:23:00 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:23:00 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:23:00 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:23:00 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:23:00 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:23:00 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:00 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:23:00 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:03 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:23:03 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:23:03 oak-gw06 kernel: CPU: 4 PID: 11877 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:23:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:23:03 oak-gw06 kernel: 0000000000104020 000000000978b0b5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:23:03 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:23:03 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff880426f92000 000000000978b0b5 Aug 1 10:23:03 oak-gw06 kernel: Call Trace: Aug 1 10:23:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:23:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:23:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:23:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:23:03 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:23:03 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:23:03 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:23:03 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:23:03 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:23:03 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:23:03 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:23:03 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:23:03 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:23:03 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:23:03 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:23:03 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:23:03 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:23:03 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:23:03 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:23:03 oak-gw06 kernel: [] ? memcpy_toiovec+0x4a/0x90 Aug 1 10:23:03 oak-gw06 kernel: [] skb_copy_datagram_iovec+0x5b/0x2a0 Aug 1 10:23:03 oak-gw06 kernel: [] ? kmem_cache_free+0x1bb/0x1f0 Aug 1 10:23:03 oak-gw06 kernel: [] tcp_recvmsg+0x24a/0xb50 Aug 1 10:23:03 oak-gw06 kernel: [] ? osc_io_iter_fini+0x65/0xd0 [osc] Aug 1 10:23:03 oak-gw06 kernel: [] inet_recvmsg+0x7b/0xa0 Aug 1 10:23:03 oak-gw06 kernel: [] sock_recvmsg+0xbf/0x100 Aug 1 10:23:03 oak-gw06 kernel: [] SYSC_recvfrom+0xe8/0x160 Aug 1 10:23:03 oak-gw06 kernel: [] ? poll_select_copy_remaining+0xfc/0x150 Aug 1 10:23:03 oak-gw06 kernel: [] SyS_recvfrom+0xe/0x10 Aug 1 10:23:03 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:23:03 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:23:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:23:03 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:23:03 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000046082a7fd630 0000000000000001 Aug 1 10:23:03 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa3e00 00000000210cf37f Aug 1 10:23:03 oak-gw06 kernel: Call Trace: Aug 1 10:23:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:23:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:23:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:23:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:23:03 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:23:03 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:23:03 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:23:03 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:23:03 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:23:03 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:23:03 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:23:03 oak-gw06 kernel: [] ? sched_clock+0x9/0x10 Aug 1 10:23:03 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:23:03 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:23:03 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:23:03 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:23:03 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:23:03 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:23:03 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:23:03 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:23:03 oak-gw06 kernel: [] ? put_page+0x1a/0x60 Aug 1 10:23:03 oak-gw06 kernel: [] delete_from_page_cache+0x6c/0xa0 Aug 1 10:23:03 oak-gw06 kernel: [] vvp_page_discard+0xa2/0x160 [lustre] Aug 1 10:23:03 oak-gw06 kernel: [] cl_page_invoid+0x68/0x170 [obdclass] Aug 1 10:23:03 oak-gw06 kernel: [] ? cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:23:03 oak-gw06 kernel: [] cl_page_discard+0x13/0x20 [obdclass] Aug 1 10:23:03 oak-gw06 kernel: [] discard_pagevec+0x60/0xd0 [osc] Aug 1 10:23:03 oak-gw06 kernel: [] osc_lru_shrink+0x3f7/0x750 [osc] Aug 1 10:23:03 oak-gw06 kernel: [] ? cl_env_get+0x1bb/0x270 [obdclass] Aug 1 10:23:03 oak-gw06 kernel: [] osc_cache_shrink_scan+0x143/0x190 [osc] Aug 1 10:23:03 oak-gw06 kernel: [] osc_cache_shrink+0x36/0x60 [osc] Aug 1 10:23:03 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:23:03 oak-gw06 kernel: [] ? vmpressure+0x61/0x90 Aug 1 10:23:03 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:23:03 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:23:03 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:23:03 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:23:03 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:23:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:03 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:23:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:23:49 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:23:49 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:23:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:23:49 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:23:49 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8800109b4050 ffff880415b500b8 Aug 1 10:23:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa45c0 00000000210cf37f Aug 1 10:23:49 oak-gw06 kernel: Call Trace: Aug 1 10:23:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:23:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:23:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:23:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:23:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:23:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:23:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:23:49 oak-gw06 kernel: [] ? lovsub_object_free+0x131/0x3b0 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] ? _raw_spin_lock+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_cache_writeback_range+0x90/0x1260 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_slice_add+0x5c/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_io_init+0x81/0x140 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_init0.isra.15+0x88/0x160 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] osc_io_fsync_start+0x88/0x3c0 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_req_attr_set+0x150/0x150 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_start+0x56/0x150 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_sync_file_range+0x2db/0x380 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ll_delete_inode+0xa6/0x1c0 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:23:49 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:23:49 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:23:49 oak-gw06 kernel: [] ? vmpressure+0x61/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:23:49 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:23:49 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:23:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:23:49 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:23:49 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8800109b4050 ffff880415b500b8 Aug 1 10:23:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa45c0 00000000210cf37f Aug 1 10:23:49 oak-gw06 kernel: Call Trace: Aug 1 10:23:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:23:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:23:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:23:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:23:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:23:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:23:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:23:49 oak-gw06 kernel: [] ? lovsub_object_free+0x131/0x3b0 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] ? _raw_spin_lock+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_cache_writeback_range+0x90/0x1260 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_slice_add+0x5c/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_io_init+0x81/0x140 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_init0.isra.15+0x88/0x160 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] osc_io_fsync_start+0x88/0x3c0 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_req_attr_set+0x150/0x150 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_start+0x56/0x150 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_sync_file_range+0x2db/0x380 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ll_delete_inode+0xa6/0x1c0 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:23:49 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:23:49 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:23:49 oak-gw06 kernel: [] ? vmpressure+0x61/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:23:49 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:23:49 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:23:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:23:49 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:23:49 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8800109b4050 ffff880415b500b8 Aug 1 10:23:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa45c0 00000000210cf37f Aug 1 10:23:49 oak-gw06 kernel: Call Trace: Aug 1 10:23:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:23:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:23:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:23:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:23:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:23:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:23:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:23:49 oak-gw06 kernel: [] ? lovsub_object_free+0x131/0x3b0 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] ? _raw_spin_lock+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_cache_writeback_range+0x90/0x1260 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_slice_add+0x5c/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_io_init+0x81/0x140 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_init0.isra.15+0x88/0x160 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] osc_io_fsync_start+0x88/0x3c0 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_req_attr_set+0x150/0x150 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_start+0x56/0x150 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_sync_file_range+0x2db/0x380 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ll_delete_inode+0xa6/0x1c0 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:23:49 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:23:49 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:23:49 oak-gw06 kernel: [] ? vmpressure+0x61/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:23:49 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:23:49 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:23:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:23:49 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:23:49 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8800109b4050 ffff880415b500b8 Aug 1 10:23:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa45c0 00000000210cf37f Aug 1 10:23:49 oak-gw06 kernel: Call Trace: Aug 1 10:23:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:23:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:23:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:23:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:23:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:23:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:23:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:23:49 oak-gw06 kernel: [] ? lovsub_object_free+0x131/0x3b0 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] ? _raw_spin_lock+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_cache_writeback_range+0x90/0x1260 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_slice_add+0x5c/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_io_init+0x81/0x140 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_init0.isra.15+0x88/0x160 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] osc_io_fsync_start+0x88/0x3c0 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_req_attr_set+0x150/0x150 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_start+0x56/0x150 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_sync_file_range+0x2db/0x380 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ll_delete_inode+0xa6/0x1c0 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:23:49 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:23:49 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:23:49 oak-gw06 kernel: [] ? vmpressure+0x61/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:23:49 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:23:49 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:23:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:23:49 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:23:49 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8800109b4050 ffff880415b500b8 Aug 1 10:23:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa45c0 00000000210cf37f Aug 1 10:23:49 oak-gw06 kernel: Call Trace: Aug 1 10:23:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:23:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:23:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:23:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:23:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:23:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:23:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:23:49 oak-gw06 kernel: [] ? lovsub_object_free+0x131/0x3b0 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] ? _raw_spin_lock+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_cache_writeback_range+0x90/0x1260 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_slice_add+0x5c/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_io_init+0x81/0x140 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_init0.isra.15+0x88/0x160 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] osc_io_fsync_start+0x88/0x3c0 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_req_attr_set+0x150/0x150 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_start+0x56/0x150 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_sync_file_range+0x2db/0x380 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ll_delete_inode+0xa6/0x1c0 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:23:49 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:23:49 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:23:49 oak-gw06 kernel: [] ? vmpressure+0x61/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:23:49 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:23:49 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:23:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:23:49 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:23:49 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8800109b4050 ffff880415b500b8 Aug 1 10:23:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa45c0 00000000210cf37f Aug 1 10:23:49 oak-gw06 kernel: Call Trace: Aug 1 10:23:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:23:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:23:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:23:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:23:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:23:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:23:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:23:49 oak-gw06 kernel: [] ? lovsub_object_free+0x131/0x3b0 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] ? _raw_spin_lock+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_cache_writeback_range+0x90/0x1260 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_slice_add+0x5c/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_io_init+0x81/0x140 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_init0.isra.15+0x88/0x160 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] osc_io_fsync_start+0x88/0x3c0 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_req_attr_set+0x150/0x150 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_start+0x56/0x150 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_sync_file_range+0x2db/0x380 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ll_delete_inode+0xa6/0x1c0 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:23:49 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:23:49 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:23:49 oak-gw06 kernel: [] ? vmpressure+0x61/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:23:49 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:23:49 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:23:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:23:49 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:23:49 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8800109b4050 ffff880415b500b8 Aug 1 10:23:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa45c0 00000000210cf37f Aug 1 10:23:49 oak-gw06 kernel: Call Trace: Aug 1 10:23:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:23:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:23:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:23:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:23:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:23:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:23:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:23:49 oak-gw06 kernel: [] ? lovsub_object_free+0x131/0x3b0 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] ? _raw_spin_lock+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_cache_writeback_range+0x90/0x1260 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_slice_add+0x5c/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_io_init+0x81/0x140 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_init0.isra.15+0x88/0x160 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] osc_io_fsync_start+0x88/0x3c0 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_req_attr_set+0x150/0x150 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_start+0x56/0x150 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_sync_file_range+0x2db/0x380 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ll_delete_inode+0xa6/0x1c0 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:23:49 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:23:49 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:23:49 oak-gw06 kernel: [] ? vmpressure+0x61/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:23:49 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:23:49 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:23:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:23:49 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:23:49 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8800109b4050 ffff880415b500b8 Aug 1 10:23:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa45c0 00000000210cf37f Aug 1 10:23:49 oak-gw06 kernel: Call Trace: Aug 1 10:23:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:23:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:23:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:23:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:23:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:23:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:23:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:23:49 oak-gw06 kernel: [] ? lovsub_object_free+0x131/0x3b0 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] ? _raw_spin_lock+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_cache_writeback_range+0x90/0x1260 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_slice_add+0x5c/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_io_init+0x81/0x140 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_init0.isra.15+0x88/0x160 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] osc_io_fsync_start+0x88/0x3c0 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_req_attr_set+0x150/0x150 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_start+0x56/0x150 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_sync_file_range+0x2db/0x380 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ll_delete_inode+0xa6/0x1c0 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:23:49 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:23:49 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:23:49 oak-gw06 kernel: [] ? vmpressure+0x61/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:23:49 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: kswapd0: page allocation failure: order:2, mode:0x104020 Aug 1 10:23:49 oak-gw06 kernel: CPU: 3 PID: 60 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:23:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:23:49 oak-gw06 kernel: 0000000000104020 00000000210cf37f ffff88043fcc39d8 ffffffff8168662f Aug 1 10:23:49 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff8800109b4050 ffff880415b500b8 Aug 1 10:23:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa45c0 00000000210cf37f Aug 1 10:23:49 oak-gw06 kernel: Call Trace: Aug 1 10:23:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:23:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:23:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:23:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:23:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:23:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:23:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:23:49 oak-gw06 kernel: [] ? lovsub_object_free+0x131/0x3b0 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] ? _raw_spin_lock+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_cache_writeback_range+0x90/0x1260 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_slice_add+0x5c/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] ? osc_io_init+0x81/0x140 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_io_init0.isra.15+0x88/0x160 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] osc_io_fsync_start+0x88/0x3c0 [osc] Aug 1 10:23:49 oak-gw06 kernel: [] ? cl_req_attr_set+0x150/0x150 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] lov_io_start+0x56/0x150 [lov] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:23:49 oak-gw06 kernel: [] cl_sync_file_range+0x2db/0x380 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ll_delete_inode+0xa6/0x1c0 [lustre] Aug 1 10:23:49 oak-gw06 kernel: [] ? inode_wait_for_writeback+0x2e/0x40 Aug 1 10:23:49 oak-gw06 kernel: [] evict+0xa7/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] dispose_list+0x3e/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] prune_icache_sb+0x163/0x320 Aug 1 10:23:49 oak-gw06 kernel: [] prune_super+0x143/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] shrink_slab+0x163/0x330 Aug 1 10:23:49 oak-gw06 kernel: [] ? vmpressure+0x61/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] balance_pgdat+0x4b1/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kswapd+0x173/0x450 Aug 1 10:23:49 oak-gw06 kernel: [] ? wake_up_atomic_t+0x30/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] ? balance_pgdat+0x5e0/0x5e0 Aug 1 10:23:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:23:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:23:49 oak-gw06 kernel: CPU: 4 PID: 11877 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:23:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:23:49 oak-gw06 kernel: 0000000000104020 000000000978b0b5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:23:49 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:23:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9f00 000000000978b0b5 Aug 1 10:23:49 oak-gw06 kernel: Call Trace: Aug 1 10:23:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:23:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:23:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:23:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:23:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:23:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:23:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:23:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:23:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:23:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:23:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:23:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:23:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:23:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:24:38 oak-gw06 kernel: warn_alloc_failed: 197 callbacks suppressed Aug 1 10:24:38 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:24:38 oak-gw06 kernel: CPU: 4 PID: 11875 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:24:38 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:24:38 oak-gw06 kernel: Aug 1 10:24:38 oak-gw06 kernel: 0000000000104020 Aug 1 10:24:38 oak-gw06 kernel: 00000000ee72dfbe ffff88043fd039f8 ffffffff8168662f Aug 1 10:24:38 oak-gw06 kernel: ptlrpcd_00_01: page allocation failure: order:2, mode:0x104020 Aug 1 10:24:38 oak-gw06 kernel: CPU: 3 PID: 1763 Comm: ptlrpcd_00_01 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:24:38 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:24:38 oak-gw06 kernel: 0000000000104020 00000000e73c1259 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:24:38 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:24:38 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5d00 00000000e73c1259 Aug 1 10:24:38 oak-gw06 kernel: Call Trace: Aug 1 10:24:38 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:24:38 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:24:38 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:24:38 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:24:38 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:24:38 oak-gw06 kernel: [] ? __dev_kfree_skb_any+0x3d/0x50 Aug 1 10:24:38 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:24:38 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:24:38 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:24:38 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:24:38 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:24:38 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:24:38 oak-gw06 kernel: [] ? loop_64+0x21/0x78 [crc32_pclmul] Aug 1 10:24:38 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:24:38 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:24:38 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:24:38 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:24:38 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:24:38 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:24:38 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:24:38 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:24:38 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:24:38 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:24:38 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:24:38 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:24:38 oak-gw06 kernel: ptlrpcd_00_01: page allocation failure: order:2, mode:0x104020 Aug 1 10:24:38 oak-gw06 kernel: CPU: 3 PID: 1763 Comm: ptlrpcd_00_01 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:24:38 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:24:38 oak-gw06 kernel: 0000000000104020 00000000e73c1259 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:24:38 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:24:38 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5d00 00000000e73c1259 Aug 1 10:24:38 oak-gw06 kernel: Call Trace: Aug 1 10:24:38 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:24:38 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:24:38 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:24:38 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:24:38 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:24:38 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:24:38 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:24:38 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:24:38 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:24:38 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:24:38 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:24:38 oak-gw06 kernel: [] ? loop_64+0x21/0x78 [crc32_pclmul] Aug 1 10:24:38 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:24:38 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:24:38 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:24:38 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:24:38 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:24:38 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:24:38 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:24:38 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:24:38 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:24:38 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:24:38 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:24:38 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:24:38 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:24:38 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:24:38 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:24:38 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:24:38 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:24:38 oak-gw06 kernel: [] ? loop_64+0x21/0x78 [crc32_pclmul] Aug 1 10:24:38 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:24:38 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:24:38 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:24:38 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:24:38 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:24:38 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:24:38 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:24:38 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:24:38 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:24:38 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:24:38 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff88025c4eb640 ffff88042b470e00 Aug 1 10:24:38 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88043fd03a70 00000000ee72dfbe Aug 1 10:24:38 oak-gw06 kernel: Call Trace: Aug 1 10:24:38 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:24:38 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:24:38 oak-gw06 kernel: [] ? ip_local_deliver+0x59/0xd0 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:24:38 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:24:38 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:24:38 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] ? napi_gro_receive+0xd8/0x130 Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] ? handle_edge_irq+0x85/0x130 Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:24:38 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:24:38 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:24:38 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:24:38 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 1 10:24:38 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 1 10:24:38 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:24:38 oak-gw06 kernel: [] ? memcpy_toiovec+0x4a/0x90 Aug 1 10:24:38 oak-gw06 kernel: [] skb_copy_datagram_iovec+0x5b/0x2a0 Aug 1 10:24:38 oak-gw06 kernel: [] ? kmem_cache_free+0x1bb/0x1f0 Aug 1 10:24:38 oak-gw06 kernel: [] tcp_recvmsg+0x24a/0xb50 Aug 1 10:24:38 oak-gw06 kernel: [] ? osc_io_iter_fini+0x65/0xd0 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] inet_recvmsg+0x7b/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] sock_recvmsg+0xbf/0x100 Aug 1 10:24:38 oak-gw06 kernel: [] SYSC_recvfrom+0xe8/0x160 Aug 1 10:24:38 oak-gw06 kernel: [] ? poll_select_copy_remaining+0xfc/0x150 Aug 1 10:24:38 oak-gw06 kernel: [] SyS_recvfrom+0xe/0x10 Aug 1 10:24:38 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:24:38 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:24:38 oak-gw06 kernel: CPU: 4 PID: 11875 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:24:38 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:24:38 oak-gw06 kernel: 0000000000104020 00000000ee72dfbe ffff88043fd039f8 ffffffff8168662f Aug 1 10:24:38 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff88025c4eb640 ffff88042b470e00 Aug 1 10:24:38 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88043fd03a70 00000000ee72dfbe Aug 1 10:24:38 oak-gw06 kernel: Call Trace: Aug 1 10:24:38 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:24:38 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:24:38 oak-gw06 kernel: [] ? ip_local_deliver+0x59/0xd0 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:24:38 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:24:38 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:24:38 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] ? napi_gro_receive+0xd8/0x130 Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:24:38 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:24:38 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:24:38 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:24:38 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 1 10:24:38 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 1 10:24:38 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:24:38 oak-gw06 kernel: [] ? memcpy_toiovec+0x4a/0x90 Aug 1 10:24:38 oak-gw06 kernel: [] skb_copy_datagram_iovec+0x5b/0x2a0 Aug 1 10:24:38 oak-gw06 kernel: [] ? kmem_cache_free+0x1bb/0x1f0 Aug 1 10:24:38 oak-gw06 kernel: [] tcp_recvmsg+0x24a/0xb50 Aug 1 10:24:38 oak-gw06 kernel: [] ? osc_io_iter_fini+0x65/0xd0 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] inet_recvmsg+0x7b/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] sock_recvmsg+0xbf/0x100 Aug 1 10:24:38 oak-gw06 kernel: [] SYSC_recvfrom+0xe8/0x160 Aug 1 10:24:38 oak-gw06 kernel: [] ? poll_select_copy_remaining+0xfc/0x150 Aug 1 10:24:38 oak-gw06 kernel: [] SyS_recvfrom+0xe/0x10 Aug 1 10:24:38 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:24:38 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:24:38 oak-gw06 kernel: CPU: 4 PID: 11875 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:24:38 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:24:38 oak-gw06 kernel: 0000000000104020 00000000ee72dfbe ffff88043fd039f8 ffffffff8168662f Aug 1 10:24:38 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff88025c4eb640 ffff88042b470e00 Aug 1 10:24:38 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88043fd03a70 00000000ee72dfbe Aug 1 10:24:38 oak-gw06 kernel: Call Trace: Aug 1 10:24:38 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:24:38 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:24:38 oak-gw06 kernel: [] ? ip_local_deliver+0x59/0xd0 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:24:38 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:24:38 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:24:38 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] ? napi_gro_receive+0xd8/0x130 Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:24:38 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:24:38 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:24:38 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:24:38 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 1 10:24:38 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 1 10:24:38 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:24:38 oak-gw06 kernel: [] ? memcpy_toiovec+0x4a/0x90 Aug 1 10:24:38 oak-gw06 kernel: [] skb_copy_datagram_iovec+0x5b/0x2a0 Aug 1 10:24:38 oak-gw06 kernel: [] ? kmem_cache_free+0x1bb/0x1f0 Aug 1 10:24:38 oak-gw06 kernel: [] tcp_recvmsg+0x24a/0xb50 Aug 1 10:24:38 oak-gw06 kernel: [] ? osc_io_iter_fini+0x65/0xd0 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] inet_recvmsg+0x7b/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] sock_recvmsg+0xbf/0x100 Aug 1 10:24:38 oak-gw06 kernel: [] SYSC_recvfrom+0xe8/0x160 Aug 1 10:24:38 oak-gw06 kernel: [] ? poll_select_copy_remaining+0xfc/0x150 Aug 1 10:24:38 oak-gw06 kernel: [] SyS_recvfrom+0xe/0x10 Aug 1 10:24:38 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:24:38 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:24:38 oak-gw06 kernel: CPU: 4 PID: 11875 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:24:38 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:24:38 oak-gw06 kernel: 0000000000104020 00000000ee72dfbe ffff88043fd039f8 ffffffff8168662f Aug 1 10:24:38 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff88025c4eb640 ffff88042b470e00 Aug 1 10:24:38 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88043fd03a70 00000000ee72dfbe Aug 1 10:24:38 oak-gw06 kernel: Call Trace: Aug 1 10:24:38 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:24:38 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:24:38 oak-gw06 kernel: [] ? ip_local_deliver+0x59/0xd0 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:24:38 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:24:38 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:24:38 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] ? napi_gro_receive+0xd8/0x130 Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:24:38 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:24:38 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:24:38 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:24:38 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 1 10:24:38 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 1 10:24:38 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:24:38 oak-gw06 kernel: [] ? memcpy_toiovec+0x4a/0x90 Aug 1 10:24:38 oak-gw06 kernel: [] skb_copy_datagram_iovec+0x5b/0x2a0 Aug 1 10:24:38 oak-gw06 kernel: [] ? kmem_cache_free+0x1bb/0x1f0 Aug 1 10:24:38 oak-gw06 kernel: [] tcp_recvmsg+0x24a/0xb50 Aug 1 10:24:38 oak-gw06 kernel: [] ? osc_io_iter_fini+0x65/0xd0 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] inet_recvmsg+0x7b/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] sock_recvmsg+0xbf/0x100 Aug 1 10:24:38 oak-gw06 kernel: [] SYSC_recvfrom+0xe8/0x160 Aug 1 10:24:38 oak-gw06 kernel: [] ? poll_select_copy_remaining+0xfc/0x150 Aug 1 10:24:38 oak-gw06 kernel: [] SyS_recvfrom+0xe/0x10 Aug 1 10:24:38 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:24:38 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:24:38 oak-gw06 kernel: CPU: 4 PID: 11875 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:24:38 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:24:38 oak-gw06 kernel: 0000000000104020 00000000ee72dfbe ffff88043fd039f8 ffffffff8168662f Aug 1 10:24:38 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff88025c4eb640 ffff88042b470e00 Aug 1 10:24:38 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88043fd03a70 00000000ee72dfbe Aug 1 10:24:38 oak-gw06 kernel: Call Trace: Aug 1 10:24:38 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:24:38 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:24:38 oak-gw06 kernel: [] ? ip_local_deliver+0x59/0xd0 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:24:38 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:24:38 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:24:38 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] ? napi_gro_receive+0xd8/0x130 Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:24:38 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:24:38 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:24:38 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:24:38 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 1 10:24:38 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 1 10:24:38 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:24:38 oak-gw06 kernel: [] ? memcpy_toiovec+0x4a/0x90 Aug 1 10:24:38 oak-gw06 kernel: [] skb_copy_datagram_iovec+0x5b/0x2a0 Aug 1 10:24:38 oak-gw06 kernel: [] ? kmem_cache_free+0x1bb/0x1f0 Aug 1 10:24:38 oak-gw06 kernel: [] tcp_recvmsg+0x24a/0xb50 Aug 1 10:24:38 oak-gw06 kernel: [] ? osc_io_iter_fini+0x65/0xd0 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] inet_recvmsg+0x7b/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] sock_recvmsg+0xbf/0x100 Aug 1 10:24:38 oak-gw06 kernel: [] SYSC_recvfrom+0xe8/0x160 Aug 1 10:24:38 oak-gw06 kernel: [] ? poll_select_copy_remaining+0xfc/0x150 Aug 1 10:24:38 oak-gw06 kernel: [] SyS_recvfrom+0xe/0x10 Aug 1 10:24:38 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:24:38 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:24:38 oak-gw06 kernel: CPU: 4 PID: 11875 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:24:38 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:24:38 oak-gw06 kernel: 0000000000104020 00000000ee72dfbe ffff88043fd039f8 ffffffff8168662f Aug 1 10:24:38 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff88025c4eb640 ffff88042b470e00 Aug 1 10:24:38 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88043fd03a70 00000000ee72dfbe Aug 1 10:24:38 oak-gw06 kernel: Call Trace: Aug 1 10:24:38 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:24:38 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:24:38 oak-gw06 kernel: [] ? ip_local_deliver+0x59/0xd0 Aug 1 10:24:38 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:24:38 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:24:38 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:24:38 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:24:38 oak-gw06 kernel: [] ? napi_gro_receive+0xd8/0x130 Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:24:38 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:24:38 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:24:38 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:24:38 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:24:38 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 1 10:24:38 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 1 10:24:38 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:24:38 oak-gw06 kernel: [] ? memcpy_toiovec+0x4a/0x90 Aug 1 10:24:38 oak-gw06 kernel: [] skb_copy_datagram_iovec+0x5b/0x2a0 Aug 1 10:24:38 oak-gw06 kernel: [] ? kmem_cache_free+0x1bb/0x1f0 Aug 1 10:24:38 oak-gw06 kernel: [] tcp_recvmsg+0x24a/0xb50 Aug 1 10:24:38 oak-gw06 kernel: [] ? osc_io_iter_fini+0x65/0xd0 [osc] Aug 1 10:24:38 oak-gw06 kernel: [] inet_recvmsg+0x7b/0xa0 Aug 1 10:24:38 oak-gw06 kernel: [] sock_recvmsg+0xbf/0x100 Aug 1 10:24:38 oak-gw06 kernel: [] SYSC_recvfrom+0xe8/0x160 Aug 1 10:24:38 oak-gw06 kernel: [] ? poll_select_copy_remaining+0xfc/0x150 Aug 1 10:24:38 oak-gw06 kernel: [] SyS_recvfrom+0xe/0x10 Aug 1 10:24:38 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:24:39 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:24:39 oak-gw06 kernel: [] ? loop_64+0x21/0x78 [crc32_pclmul] Aug 1 10:24:39 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:24:39 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:24:39 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:24:39 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:24:39 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:24:39 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:24:39 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:24:39 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:24:39 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 1 10:24:39 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 1 10:24:39 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:24:39 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 1 10:24:39 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:24:39 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:24:39 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:24:39 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:24:39 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:24:39 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:24:39 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:24:39 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:24:39 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:24:39 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:13 oak-gw06 kernel: warn_alloc_failed: 472 callbacks suppressed Aug 1 10:25:13 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 10:25:13 oak-gw06 kernel: CPU: 1 PID: 11865 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:25:13 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:25:13 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:25:13 oak-gw06 kernel: 00000000000080d0 000000003a207b03 ffff88026f1e7858 ffffffff8168662f Aug 1 10:25:13 oak-gw06 kernel: ffff88026f1e78e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 10:25:13 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88026f1e78b8 000000003a207b03 Aug 1 10:25:13 oak-gw06 kernel: Call Trace: Aug 1 10:25:13 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:25:13 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:25:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:25:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:25:13 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 10:25:13 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 10:25:13 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:25:13 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:25:13 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:25:13 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:25:13 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:25:13 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:25:13 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:25:13 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:25:13 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:25:13 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:25:13 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:25:13 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:25:13 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:25:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:13 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:25:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:13 oak-gw06 kernel: Mem-Info: Aug 1 10:25:13 oak-gw06 kernel: active_anon:28491 inactive_anon:41519 isolated_anon:0#012 active_file:677346 inactive_file:1973499 isolated_file:0#012 unevictable:0 dirty:5996 writeback:3528 unstable:0#012 slab_reclaimable:38423 slab_unreclaimable:848047#012 mapped:5607 shmem:39000 pagetables:1639 bounce:0#012 free:313828 free_pcp:846 free_cma:0 Aug 1 10:25:13 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:25:13 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:25:13 oak-gw06 kernel: Node 0 DMA32 free:220100kB min:11976kB low:14968kB high:17964kB active_anon:16596kB inactive_anon:31248kB active_file:473732kB inactive_file:1414488kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4608kB writeback:2768kB mapped:1996kB shmem:31276kB slab_reclaimable:28200kB slab_unreclaimable:611060kB kernel_stack:1088kB pagetables:1284kB unstable:0kB bounce:0kB free_pcp:1692kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:25:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:25:13 oak-gw06 kernel: Node 0 Normal free:1014336kB min:55536kB low:69420kB high:83304kB active_anon:97368kB inactive_anon:134828kB active_file:2243540kB inactive_file:6481040kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:23600kB writeback:13284kB mapped:20432kB shmem:124724kB slab_reclaimable:125492kB slab_unreclaimable:2781112kB kernel_stack:4592kB pagetables:5272kB unstable:0kB bounce:0kB free_pcp:2476kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:25:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:25:13 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:25:13 oak-gw06 kernel: Node 0 DMA32: 1560*4kB (UE) 1694*8kB (UE) 8574*16kB (UEM) 1623*32kB (UEM) 99*64kB (UM) 3*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 215632kB Aug 1 10:25:13 oak-gw06 kernel: Node 0 Normal: 7022*4kB (UE) 7301*8kB (UE) 42339*16kB (UEM) 6823*32kB (UEM) 302*64kB (UEM) 3*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1001968kB Aug 1 10:25:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:25:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:25:13 oak-gw06 kernel: 2116755 total pagecache pages Aug 1 10:25:13 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:25:13 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:25:13 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:25:13 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:25:13 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:25:13 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:25:13 oak-gw06 kernel: 127313 pages reserved Aug 1 10:25:13 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 10:25:13 oak-gw06 kernel: CPU: 1 PID: 11865 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:25:13 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:25:13 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:25:13 oak-gw06 kernel: 00000000000080d0 000000003a207b03 ffff88026f1e7808 ffffffff8168662f Aug 1 10:25:13 oak-gw06 kernel: ffff88026f1e7898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 1 10:25:13 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88026f1e7868 000000003a207b03 Aug 1 10:25:13 oak-gw06 kernel: Call Trace: Aug 1 10:25:13 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:25:13 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:25:13 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 1 10:25:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:25:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:25:13 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:25:13 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:25:13 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 10:25:13 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 10:25:13 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:25:13 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:25:13 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:25:13 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:25:13 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:25:13 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:25:13 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:25:13 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:25:13 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:25:13 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:25:13 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:25:13 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:25:13 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:25:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:13 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:25:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:13 oak-gw06 kernel: Mem-Info: Aug 1 10:25:13 oak-gw06 kernel: active_anon:28491 inactive_anon:41519 isolated_anon:0#012 active_file:684541 inactive_file:1980253 isolated_file:0#012 unevictable:0 dirty:6567 writeback:2515 unstable:0#012 slab_reclaimable:38423 slab_unreclaimable:847979#012 mapped:5607 shmem:39000 pagetables:1639 bounce:0#012 free:299486 free_pcp:675 free_cma:0 Aug 1 10:25:13 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:25:13 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:25:13 oak-gw06 kernel: Node 0 DMA32 free:207944kB min:11976kB low:14968kB high:17964kB active_anon:16596kB inactive_anon:31248kB active_file:478996kB inactive_file:1420128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4608kB writeback:2208kB mapped:1996kB shmem:31276kB slab_reclaimable:28200kB slab_unreclaimable:611060kB kernel_stack:1088kB pagetables:1284kB unstable:0kB bounce:0kB free_pcp:1508kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:25:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:25:13 oak-gw06 kernel: Node 0 Normal free:951996kB min:55536kB low:69420kB high:83304kB active_anon:97368kB inactive_anon:134828kB active_file:2266940kB inactive_file:6506780kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:25928kB writeback:9404kB mapped:20432kB shmem:124724kB slab_reclaimable:125492kB slab_unreclaimable:2780840kB kernel_stack:4592kB pagetables:5272kB unstable:0kB bounce:0kB free_pcp:1828kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:25:13 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:25:13 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:25:13 oak-gw06 kernel: Node 0 DMA32: 1065*4kB (UEM) 1718*8kB (UE) 7980*16kB (UEM) 1639*32kB (UEM) 101*64kB (UM) 3*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 204980kB Aug 1 10:25:13 oak-gw06 kernel: Node 0 Normal: 4728*4kB (UE) 7350*8kB (UE) 39158*16kB (UEM) 6867*32kB (UEM) 312*64kB (UEM) 3*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 944336kB Aug 1 10:25:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:25:13 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:25:13 oak-gw06 kernel: 2116981 total pagecache pages Aug 1 10:25:13 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:25:13 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:25:13 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:25:13 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:25:13 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:25:13 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:25:13 oak-gw06 kernel: 127313 pages reserved Aug 1 10:25:49 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 1 10:25:49 oak-gw06 kernel: ptlrpcd_00_04: page allocation failure: order:2, mode:0x104020 Aug 1 10:25:49 oak-gw06 kernel: CPU: 4 PID: 1766 Comm: ptlrpcd_00_04 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:25:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:25:49 oak-gw06 kernel: 0000000000104020 00000000c9f1c6e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:25:49 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:25:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e8000 00000000c9f1c6e5 Aug 1 10:25:49 oak-gw06 kernel: Call Trace: Aug 1 10:25:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:25:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:25:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:25:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:25:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:25:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:25:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:25:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:25:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:25:49 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 1 10:25:49 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:25:49 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:25:49 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:25:49 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: ptlrpcd_00_04: page allocation failure: order:2, mode:0x104020 Aug 1 10:25:49 oak-gw06 kernel: CPU: 4 PID: 1766 Comm: ptlrpcd_00_04 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:25:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:25:49 oak-gw06 kernel: 0000000000104020 00000000c9f1c6e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:25:49 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:25:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9740 00000000c9f1c6e5 Aug 1 10:25:49 oak-gw06 kernel: Call Trace: Aug 1 10:25:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:25:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:25:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:25:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:25:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:25:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:25:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:25:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:25:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:25:49 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 1 10:25:49 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:25:49 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:25:49 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:25:49 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: ptlrpcd_00_04: page allocation failure: order:2, mode:0x104020 Aug 1 10:25:49 oak-gw06 kernel: CPU: 4 PID: 1766 Comm: ptlrpcd_00_04 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:25:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:25:49 oak-gw06 kernel: 0000000000104020 00000000c9f1c6e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:25:49 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:25:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9740 00000000c9f1c6e5 Aug 1 10:25:49 oak-gw06 kernel: Call Trace: Aug 1 10:25:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:25:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:25:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:25:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:25:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:25:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:25:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:25:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:25:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:25:49 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 1 10:25:49 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:25:49 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:25:49 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:25:49 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: ptlrpcd_00_04: page allocation failure: order:2, mode:0x104020 Aug 1 10:25:49 oak-gw06 kernel: CPU: 4 PID: 1766 Comm: ptlrpcd_00_04 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:25:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:25:49 oak-gw06 kernel: 0000000000104020 00000000c9f1c6e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:25:49 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:25:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9740 00000000c9f1c6e5 Aug 1 10:25:49 oak-gw06 kernel: Call Trace: Aug 1 10:25:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:25:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:25:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:25:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:25:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:25:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:25:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:25:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:25:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:25:49 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 1 10:25:49 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:25:49 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:25:49 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:25:49 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: ptlrpcd_00_04: page allocation failure: order:2, mode:0x104020 Aug 1 10:25:49 oak-gw06 kernel: CPU: 4 PID: 1766 Comm: ptlrpcd_00_04 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:25:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:25:49 oak-gw06 kernel: 0000000000104020 00000000c9f1c6e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:25:49 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:25:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9740 00000000c9f1c6e5 Aug 1 10:25:49 oak-gw06 kernel: Call Trace: Aug 1 10:25:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:25:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:25:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:25:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:25:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:25:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:25:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:25:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:25:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:25:49 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 1 10:25:49 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:25:49 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:25:49 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:25:49 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: ptlrpcd_00_04: page allocation failure: order:2, mode:0x104020 Aug 1 10:25:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:25:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:25:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:25:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:25:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:25:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:25:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:25:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:25:49 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 1 10:25:49 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:25:49 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:25:49 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:25:49 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: ptlrpcd_00_04: page allocation failure: order:2, mode:0x104020 Aug 1 10:25:49 oak-gw06 kernel: CPU: 4 PID: 1766 Comm: ptlrpcd_00_04 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:25:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:25:49 oak-gw06 kernel: 0000000000104020 00000000c9f1c6e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:25:49 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:25:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9740 00000000c9f1c6e5 Aug 1 10:25:49 oak-gw06 kernel: Call Trace: Aug 1 10:25:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:25:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:25:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:25:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:25:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:25:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:25:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:25:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:25:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:25:49 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 1 10:25:49 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:25:49 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:25:49 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:25:49 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: ptlrpcd_00_04: page allocation failure: order:2, mode:0x104020 Aug 1 10:25:49 oak-gw06 kernel: CPU: 4 PID: 1766 Comm: ptlrpcd_00_04 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:25:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:25:49 oak-gw06 kernel: 0000000000104020 00000000c9f1c6e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:25:49 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:25:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9740 00000000c9f1c6e5 Aug 1 10:25:49 oak-gw06 kernel: Call Trace: Aug 1 10:25:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:25:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:25:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:25:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:25:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:25:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:25:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:25:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:25:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:25:49 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 1 10:25:49 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:25:49 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:25:49 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:25:49 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: ptlrpcd_00_04: page allocation failure: order:2, mode:0x104020 Aug 1 10:25:49 oak-gw06 kernel: CPU: 4 PID: 1766 Comm: ptlrpcd_00_04 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:25:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:25:49 oak-gw06 kernel: 0000000000104020 00000000c9f1c6e5 ffff88043fd039d8 ffffffff8168662f Aug 1 10:25:49 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 1 10:25:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4e9740 00000000c9f1c6e5 Aug 1 10:25:49 oak-gw06 kernel: Call Trace: Aug 1 10:25:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:25:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:25:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:25:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:25:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:25:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:25:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:25:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:25:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:25:49 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 1 10:25:49 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 1 10:25:49 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 1 10:25:49 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 1 10:25:49 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 1 10:25:49 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: CPU: 3 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:25:49 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:25:49 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fcc39d8 ffffffff8168662f Aug 1 10:25:49 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff880004630050 ffff880415b500b8 Aug 1 10:25:49 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5d00 00000000da86253b Aug 1 10:25:49 oak-gw06 kernel: Call Trace: Aug 1 10:25:49 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:25:49 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:25:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:25:49 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:25:49 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] ? napi_gro_complete+0x7d/0x100 Aug 1 10:25:49 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:25:49 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:25:49 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:25:49 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:25:49 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:25:49 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:25:49 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:25:49 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:25:49 oak-gw06 kernel: [] ? _raw_spin_lock+0x32/0x50 Aug 1 10:25:49 oak-gw06 kernel: [] osc_page_delete+0xf6/0x4e0 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:25:49 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:25:49 oak-gw06 kernel: [] ll_releasepage+0xee/0x1a0 [lustre] Aug 1 10:25:49 oak-gw06 kernel: [] try_to_release_page+0x32/0x50 Aug 1 10:25:49 oak-gw06 kernel: [] shrink_page_list+0x950/0xb00 Aug 1 10:25:49 oak-gw06 kernel: [] shrink_inactive_list+0x1fa/0x630 Aug 1 10:25:49 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:25:49 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:25:49 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:25:49 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:25:49 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:25:49 oak-gw06 kernel: [] new_slab+0x295/0x320 Aug 1 10:25:49 oak-gw06 kernel: [] ___slab_alloc+0x3ac/0x4f0 Aug 1 10:25:49 oak-gw06 kernel: [] ? ptlrpc_new_bulk+0x459/0x860 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ? __kmalloc+0x1f3/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] ? null_alloc_reqbuf+0x175/0x390 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ? ptlrpc_new_bulk+0x459/0x860 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] __slab_alloc+0x40/0x5c Aug 1 10:25:49 oak-gw06 kernel: [] __kmalloc+0x1c8/0x240 Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_new_bulk+0x459/0x860 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_prep_bulk_imp+0x5d/0x180 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ? lustre_msg_set_timeout+0x27/0xa0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] osc_brw_prep_request+0x36d/0xf10 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] osc_build_rpc+0x555/0xfb0 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] osc_check_rpcs+0xe2b/0x18b0 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? kmem_cache_free+0x1bb/0x1f0 Aug 1 10:25:49 oak-gw06 kernel: [] ? osc_extent_put+0x100/0x320 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 1 10:25:49 oak-gw06 kernel: [] ? osc_brw_fini_request+0xa72/0x12f0 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] osc_io_unplug0+0xe2/0x130 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] osc_io_unplug+0x10/0x20 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] brw_interpret+0x479/0xe60 [osc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:25:49 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:25:49 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:25:49 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:25:49 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:26:37 oak-gw06 kernel: warn_alloc_failed: 1324 callbacks suppressed Aug 1 10:26:37 oak-gw06 kernel: ptlrpcd_00_00: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:37 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:37 oak-gw06 kernel: CPU: 3 PID: 11877 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:37 oak-gw06 kernel: 0000000000104020 000000000978b0b5 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:26:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:26:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa5d00 000000000978b0b5 Aug 1 10:26:37 oak-gw06 kernel: Call Trace: Aug 1 10:26:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:37 oak-gw06 kernel: [] ? cl_page_print+0xc0/0xc0 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ? lov_page_init_raid0+0x9f/0x380 [lov] Aug 1 10:26:37 oak-gw06 kernel: [] lov_page_init+0x1c/0x50 [lov] Aug 1 10:26:37 oak-gw06 kernel: [] cl_page_alloc+0x10a/0x270 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] cl_page_find+0x74/0x280 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ll_write_begin+0xe0/0x830 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Aug 1 10:26:37 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:26:37 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:26:37 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:26:37 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:26:37 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:26:37 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:37 oak-gw06 kernel: CPU: 3 PID: 11877 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:37 oak-gw06 kernel: 0000000000104020 000000000978b0b5 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:26:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffff815ce55f 00fafd0c7fc03700 Aug 1 10:26:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000000 000000000978b0b5 Aug 1 10:26:37 oak-gw06 kernel: Call Trace: Aug 1 10:26:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:37 oak-gw06 kernel: [] ? tcp_transmit_skb+0x4af/0x990 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] ? napi_gro_complete+0x7d/0x100 Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:37 oak-gw06 kernel: [] ? cl_page_assume+0x77/0x200 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:26:37 oak-gw06 kernel: [] ? iov_iter_copy_from_user_atomic+0x6a/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] generic_file_buffered_write+0x155/0x2a0 Aug 1 10:26:37 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:26:37 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:26:37 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:26:37 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:26:37 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:26:37 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:37 oak-gw06 kernel: CPU: 3 PID: 11877 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:37 oak-gw06 kernel: 0000000000104020 000000000978b0b5 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:26:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffff815ce55f 00fafd0c7fc03700 Aug 1 10:26:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000000 000000000978b0b5 Aug 1 10:26:37 oak-gw06 kernel: Call Trace: Aug 1 10:26:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:37 oak-gw06 kernel: [] ? tcp_transmit_skb+0x4af/0x990 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:37 oak-gw06 kernel: [] ? cl_page_assume+0x77/0x200 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:26:37 oak-gw06 kernel: [] ? iov_iter_copy_from_user_atomic+0x6a/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] generic_file_buffered_write+0x155/0x2a0 Aug 1 10:26:37 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:26:37 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:26:37 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:26:37 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:26:37 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:26:37 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:37 oak-gw06 kernel: CPU: 3 PID: 11877 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:37 oak-gw06 kernel: 0000000000104020 000000000978b0b5 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:26:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffff815ce55f 00fafd0c7fc03700 Aug 1 10:26:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000000 000000000978b0b5 Aug 1 10:26:37 oak-gw06 kernel: Call Trace: Aug 1 10:26:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:37 oak-gw06 kernel: [] ? tcp_transmit_skb+0x4af/0x990 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:37 oak-gw06 kernel: [] ? cl_page_assume+0x77/0x200 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:26:37 oak-gw06 kernel: [] ? iov_iter_copy_from_user_atomic+0x6a/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] generic_file_buffered_write+0x155/0x2a0 Aug 1 10:26:37 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:26:37 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:26:37 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:26:37 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:26:37 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:26:37 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:37 oak-gw06 kernel: CPU: 3 PID: 11877 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:37 oak-gw06 kernel: 0000000000104020 000000000978b0b5 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:26:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffff815ce55f 00fafd0c7fc03700 Aug 1 10:26:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000000 000000000978b0b5 Aug 1 10:26:37 oak-gw06 kernel: Call Trace: Aug 1 10:26:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:37 oak-gw06 kernel: [] ? tcp_transmit_skb+0x4af/0x990 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:37 oak-gw06 kernel: [] ? cl_page_assume+0x77/0x200 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:26:37 oak-gw06 kernel: [] ? iov_iter_copy_from_user_atomic+0x6a/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] generic_file_buffered_write+0x155/0x2a0 Aug 1 10:26:37 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:26:37 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:26:37 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:26:37 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:26:37 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:26:37 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:37 oak-gw06 kernel: CPU: 3 PID: 11877 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:37 oak-gw06 kernel: 0000000000104020 000000000978b0b5 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:26:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffff815ce55f 00fafd0c7fc03700 Aug 1 10:26:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000000 000000000978b0b5 Aug 1 10:26:37 oak-gw06 kernel: Call Trace: Aug 1 10:26:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:37 oak-gw06 kernel: [] ? tcp_transmit_skb+0x4af/0x990 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:37 oak-gw06 kernel: [] ? cl_page_assume+0x77/0x200 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:26:37 oak-gw06 kernel: [] ? iov_iter_copy_from_user_atomic+0x6a/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] generic_file_buffered_write+0x155/0x2a0 Aug 1 10:26:37 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:26:37 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:26:37 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:26:37 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:26:37 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:26:37 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:37 oak-gw06 kernel: CPU: 3 PID: 11877 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:37 oak-gw06 kernel: 0000000000104020 000000000978b0b5 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:26:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffff815ce55f 00fafd0c7fc03700 Aug 1 10:26:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000000 000000000978b0b5 Aug 1 10:26:37 oak-gw06 kernel: Call Trace: Aug 1 10:26:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:37 oak-gw06 kernel: [] ? tcp_transmit_skb+0x4af/0x990 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:37 oak-gw06 kernel: [] ? cl_page_assume+0x77/0x200 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:26:37 oak-gw06 kernel: [] ? iov_iter_copy_from_user_atomic+0x6a/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] generic_file_buffered_write+0x155/0x2a0 Aug 1 10:26:37 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:26:37 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:26:37 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:26:37 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:26:37 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:26:37 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:37 oak-gw06 kernel: CPU: 3 PID: 11877 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:37 oak-gw06 kernel: 0000000000104020 000000000978b0b5 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:26:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffff815ce55f 00fafd0c7fc03700 Aug 1 10:26:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000000 000000000978b0b5 Aug 1 10:26:37 oak-gw06 kernel: Call Trace: Aug 1 10:26:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:37 oak-gw06 kernel: [] ? tcp_transmit_skb+0x4af/0x990 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:37 oak-gw06 kernel: [] ? cl_page_assume+0x77/0x200 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:26:37 oak-gw06 kernel: [] ? iov_iter_copy_from_user_atomic+0x6a/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] generic_file_buffered_write+0x155/0x2a0 Aug 1 10:26:37 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:26:37 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:26:37 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:26:37 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:26:37 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:26:37 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:37 oak-gw06 kernel: CPU: 3 PID: 11877 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:37 oak-gw06 kernel: 0000000000104020 000000000978b0b5 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:26:37 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffffffff815ce55f 00fafd0c7fc03700 Aug 1 10:26:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000000 000000000978b0b5 Aug 1 10:26:37 oak-gw06 kernel: Call Trace: Aug 1 10:26:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:37 oak-gw06 kernel: [] ? tcp_transmit_skb+0x4af/0x990 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:37 oak-gw06 kernel: [] ? cl_page_assume+0x77/0x200 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:26:37 oak-gw06 kernel: [] ? iov_iter_copy_from_user_atomic+0x6a/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] generic_file_buffered_write+0x155/0x2a0 Aug 1 10:26:37 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 1 10:26:37 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 1 10:26:37 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 1 10:26:37 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 1 10:26:37 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:26:37 oak-gw06 kernel: CPU: 4 PID: 1762 Comm: ptlrpcd_00_00 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:37 oak-gw06 kernel: 0000000000104020 0000000072c35f16 ffff88043fd039d8 ffffffff8168662f Aug 1 10:26:37 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 0000000000000000 0000000000000002 Aug 1 10:26:37 oak-gw06 kernel: fffffffffffffffc 0010402000000000 000000015c4e9740 0000000072c35f16 Aug 1 10:26:37 oak-gw06 kernel: Call Trace: Aug 1 10:26:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:37 oak-gw06 kernel: [] ? __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:37 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] ? swiotlb_map_page+0x4a/0x140 Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] ? swiotlb_unmap_page+0x9/0x10 Aug 1 10:26:37 oak-gw06 kernel: [] ? bnx2x_rx_int+0x279/0x17b0 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:37 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:37 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:37 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:37 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:37 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:37 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:37 oak-gw06 kernel: [] ? __list_del_entry+0x29/0xd0 Aug 1 10:26:37 oak-gw06 kernel: [] __osc_lru_del+0x2f/0x80 [osc] Aug 1 10:26:37 oak-gw06 kernel: [] osc_page_delete+0x115/0x4e0 [osc] Aug 1 10:26:37 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 1 10:26:37 oak-gw06 kernel: [] ll_releasepage+0xee/0x1a0 [lustre] Aug 1 10:26:37 oak-gw06 kernel: [] try_to_release_page+0x32/0x50 Aug 1 10:26:37 oak-gw06 kernel: [] shrink_page_list+0x950/0xb00 Aug 1 10:26:37 oak-gw06 kernel: [] shrink_inactive_list+0x1fa/0x630 Aug 1 10:26:37 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 1 10:26:37 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 1 10:26:37 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 1 10:26:37 oak-gw06 kernel: [] ? zone_statistics+0x89/0xa0 Aug 1 10:26:37 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 1 10:26:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:37 oak-gw06 kernel: [] new_slab+0x295/0x320 Aug 1 10:26:37 oak-gw06 kernel: [] ___slab_alloc+0x3ac/0x4f0 Aug 1 10:26:37 oak-gw06 kernel: [] ? ptlrpc_new_bulk+0x459/0x860 [ptlrpc] Aug 1 10:26:37 oak-gw06 kernel: [] ? __kmalloc+0x1f3/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] ? null_alloc_reqbuf+0x175/0x390 [ptlrpc] Aug 1 10:26:37 oak-gw06 kernel: [] ? ptlrpc_new_bulk+0x459/0x860 [ptlrpc] Aug 1 10:26:37 oak-gw06 kernel: [] __slab_alloc+0x40/0x5c Aug 1 10:26:37 oak-gw06 kernel: [] __kmalloc+0x1c8/0x240 Aug 1 10:26:37 oak-gw06 kernel: [] ptlrpc_new_bulk+0x459/0x860 [ptlrpc] Aug 1 10:26:37 oak-gw06 kernel: [] ptlrpc_prep_bulk_imp+0x5d/0x180 [ptlrpc] Aug 1 10:26:37 oak-gw06 kernel: [] ? lustre_msg_set_timeout+0x27/0xa0 [ptlrpc] Aug 1 10:26:37 oak-gw06 kernel: [] osc_brw_prep_request+0x36d/0xf10 [osc] Aug 1 10:26:37 oak-gw06 kernel: [] osc_build_rpc+0x555/0xfb0 [osc] Aug 1 10:26:37 oak-gw06 kernel: [] osc_check_rpcs+0xe2b/0x18b0 [osc] Aug 1 10:26:37 oak-gw06 kernel: [] ? kmem_cache_free+0x1bb/0x1f0 Aug 1 10:26:37 oak-gw06 kernel: [] ? osc_extent_put+0x100/0x320 [osc] Aug 1 10:26:37 oak-gw06 kernel: [] ? osc_extent_finish+0x606/0xb10 [osc] Aug 1 10:26:37 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 1 10:26:37 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 1 10:26:37 oak-gw06 kernel: [] osc_io_unplug0+0xe2/0x130 [osc] Aug 1 10:26:37 oak-gw06 kernel: [] osc_io_unplug+0x10/0x20 [osc] Aug 1 10:26:37 oak-gw06 kernel: [] brw_queue_work+0x31/0xd0 [osc] Aug 1 10:26:37 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 1 10:26:37 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 1 10:26:37 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 1 10:26:37 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 1 10:26:37 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 1 10:26:37 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 1 10:26:37 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 1 10:26:37 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:26:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:26:37 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:26:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:26:45 oak-gw06 kernel: warn_alloc_failed: 2377 callbacks suppressed Aug 1 10:26:45 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:45 oak-gw06 kernel: swapper/4: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:45 oak-gw06 kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:45 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:45 oak-gw06 kernel: 0000000000104020 803b52a30d0a83c1 ffff88043fd039d8 ffffffff8168662f Aug 1 10:26:45 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 0000000000000010 0000000000000000 Aug 1 10:26:45 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88025c4ec5c0 803b52a30d0a83c1 Aug 1 10:26:45 oak-gw06 kernel: Call Trace: Aug 1 10:26:45 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:46 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 1 10:26:46 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 1 10:26:46 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 1 10:26:46 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 1 10:26:46 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 1 10:26:46 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:46 oak-gw06 kernel: CPU: 4 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:46 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fd039f8 ffffffff8168662f Aug 1 10:26:46 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff8803169fcf00 ffff88025c4e8f80 Aug 1 10:26:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8803169fd000 00000000282396d3 Aug 1 10:26:46 oak-gw06 kernel: Call Trace: Aug 1 10:26:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:46 oak-gw06 kernel: [] ? tcp_v4_do_rcv+0x10a/0x340 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:26:46 oak-gw06 kernel: [] ? __dev_kfree_skb_any+0x3d/0x50 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:46 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 1 10:26:46 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 1 10:26:46 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:26:46 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:46 oak-gw06 kernel: CPU: 4 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:46 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fd039f8 ffffffff8168662f Aug 1 10:26:46 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff88043fd03aa0 ffffffff815d720c Aug 1 10:26:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffffffff81aebbd0 00000000282396d3 Aug 1 10:26:46 oak-gw06 kernel: Call Trace: Aug 1 10:26:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] ? tcp_v4_rcv+0x7ac/0x9a0 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:46 oak-gw06 kernel: [] ? ip_local_deliver_finish+0xb4/0x1f0 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:46 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 1 10:26:46 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 1 10:26:46 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:26:46 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:46 oak-gw06 kernel: CPU: 4 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:46 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fd039f8 ffffffff8168662f Aug 1 10:26:46 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff88043fd03aa0 ffffffff815d720c Aug 1 10:26:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffffffff81aebbd0 00000000282396d3 Aug 1 10:26:46 oak-gw06 kernel: Call Trace: Aug 1 10:26:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] ? tcp_v4_rcv+0x7ac/0x9a0 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:46 oak-gw06 kernel: [] ? ip_local_deliver_finish+0xb4/0x1f0 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:46 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 1 10:26:46 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 1 10:26:46 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:26:46 oak-gw06 kernel: systemd-journal: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:46 oak-gw06 kernel: CPU: 4 PID: 425 Comm: systemd-journal Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:46 oak-gw06 kernel: 0000000000104020 00000000282396d3 ffff88043fd039f8 ffffffff8168662f Aug 1 10:26:46 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff88043fd03aa0 ffffffff815d720c Aug 1 10:26:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffffffff81aebbd0 00000000282396d3 Aug 1 10:26:46 oak-gw06 kernel: Call Trace: Aug 1 10:26:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] ? tcp_v4_rcv+0x7ac/0x9a0 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:46 oak-gw06 kernel: [] ? ip_local_deliver_finish+0xb4/0x1f0 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: CPU: 3 PID: 11876 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:46 oak-gw06 kernel: 0000000000104020 00000000042ee960 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:26:46 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:26:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa3e00 00000000042ee960 Aug 1 10:26:46 oak-gw06 kernel: Call Trace: Aug 1 10:26:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:46 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:26:46 oak-gw06 kernel: [] ? file_read_actor+0x133/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] generic_file_aio_read+0x48b/0x790 Aug 1 10:26:46 oak-gw06 kernel: [] ll_file_aio_read+0x1cd/0x3e0 [lustre] Aug 1 10:26:46 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 1 10:26:46 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 1 10:26:46 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:26:46 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:46 oak-gw06 kernel: CPU: 3 PID: 11876 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:46 oak-gw06 kernel: 0000000000104020 00000000042ee960 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:26:46 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:26:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa3e00 00000000042ee960 Aug 1 10:26:46 oak-gw06 kernel: Call Trace: Aug 1 10:26:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:46 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:26:46 oak-gw06 kernel: [] ? file_read_actor+0x133/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] generic_file_aio_read+0x48b/0x790 Aug 1 10:26:46 oak-gw06 kernel: [] ll_file_aio_read+0x1cd/0x3e0 [lustre] Aug 1 10:26:46 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 1 10:26:46 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 1 10:26:46 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:26:46 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:46 oak-gw06 kernel: CPU: 3 PID: 11876 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:46 oak-gw06 kernel: 0000000000104020 00000000042ee960 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:26:46 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:26:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa3e00 00000000042ee960 Aug 1 10:26:46 oak-gw06 kernel: Call Trace: Aug 1 10:26:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:46 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:26:46 oak-gw06 kernel: [] ? file_read_actor+0x133/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] generic_file_aio_read+0x48b/0x790 Aug 1 10:26:46 oak-gw06 kernel: [] ll_file_aio_read+0x1cd/0x3e0 [lustre] Aug 1 10:26:46 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 1 10:26:46 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 1 10:26:46 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:26:46 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:46 oak-gw06 kernel: CPU: 3 PID: 11876 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:46 oak-gw06 kernel: 0000000000104020 00000000042ee960 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:26:46 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:26:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa3e00 00000000042ee960 Aug 1 10:26:46 oak-gw06 kernel: Call Trace: Aug 1 10:26:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:46 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:26:46 oak-gw06 kernel: [] ? file_read_actor+0x133/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] generic_file_aio_read+0x48b/0x790 Aug 1 10:26:46 oak-gw06 kernel: [] ll_file_aio_read+0x1cd/0x3e0 [lustre] Aug 1 10:26:46 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 1 10:26:46 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 1 10:26:46 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:26:46 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 1 10:26:46 oak-gw06 kernel: CPU: 3 PID: 11876 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:26:46 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:26:46 oak-gw06 kernel: 0000000000104020 00000000042ee960 ffff88043fcc39d8 ffffffff8168662f Aug 1 10:26:46 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 1 10:26:46 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa3e00 00000000042ee960 Aug 1 10:26:46 oak-gw06 kernel: Call Trace: Aug 1 10:26:46 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:26:46 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:26:46 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:26:46 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:26:46 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] ? task_tick_fair+0x234/0x680 Aug 1 10:26:46 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 1 10:26:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:46 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 1 10:26:46 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 1 10:26:46 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 1 10:26:46 oak-gw06 kernel: [] ? file_read_actor+0x133/0x180 Aug 1 10:26:46 oak-gw06 kernel: [] generic_file_aio_read+0x48b/0x790 Aug 1 10:26:46 oak-gw06 kernel: [] ll_file_aio_read+0x1cd/0x3e0 [lustre] Aug 1 10:26:46 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 1 10:26:46 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 1 10:26:46 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 1 10:26:46 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 1 10:26:46 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 1 10:26:46 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 1 10:26:46 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 1 10:26:46 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 1 10:26:46 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 1 10:26:46 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 1 10:26:46 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 1 10:26:46 oak-gw06 kernel: [] ? system_call_fastpath+0x16/0x1b Aug 1 10:30:14 oak-gw06 kernel: warn_alloc_failed: 159 callbacks suppressed Aug 1 10:30:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 10:30:14 oak-gw06 kernel: CPU: 2 PID: 11865 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:30:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:30:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:30:14 oak-gw06 kernel: 00000000000080d0 000000003a207b03 ffff88026f1e7858 ffffffff8168662f Aug 1 10:30:14 oak-gw06 kernel: ffff88026f1e78e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 10:30:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88026f1e78b8 000000003a207b03 Aug 1 10:30:14 oak-gw06 kernel: Call Trace: Aug 1 10:30:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:30:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:30:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:30:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:30:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 10:30:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 10:30:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:30:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:30:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:30:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:30:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:30:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:30:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:30:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:30:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:30:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:30:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:30:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:30:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:30:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:30:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:30:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:30:14 oak-gw06 kernel: Mem-Info: Aug 1 10:30:14 oak-gw06 kernel: active_anon:29271 inactive_anon:41519 isolated_anon:0#012 active_file:2048882 inactive_file:12576 isolated_file:0#012 unevictable:0 dirty:22980 writeback:6750 unstable:0#012 slab_reclaimable:37113 slab_unreclaimable:804974#012 mapped:5746 shmem:39000 pagetables:1646 bounce:0#012 free:892706 free_pcp:417 free_cma:0 Aug 1 10:30:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:30:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:30:14 oak-gw06 kernel: Node 0 DMA32 free:653016kB min:11976kB low:14968kB high:17964kB active_anon:16940kB inactive_anon:31248kB active_file:1436444kB inactive_file:7796kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:15916kB writeback:5572kB mapped:1996kB shmem:31276kB slab_reclaimable:27432kB slab_unreclaimable:579488kB kernel_stack:1040kB pagetables:1116kB unstable:0kB bounce:0kB free_pcp:508kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:30:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:30:14 oak-gw06 kernel: Node 0 Normal free:2893584kB min:55536kB low:69420kB high:83304kB active_anon:100144kB inactive_anon:134828kB active_file:6770468kB inactive_file:33956kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:78676kB writeback:21428kB mapped:20988kB shmem:124724kB slab_reclaimable:121020kB slab_unreclaimable:2640392kB kernel_stack:4624kB pagetables:5468kB unstable:0kB bounce:0kB free_pcp:1596kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:30:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:30:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:30:14 oak-gw06 kernel: Node 0 DMA32: 230*4kB (UEM) 1277*8kB (UEM) 13507*16kB (UEM) 10435*32kB (UEM) 1328*64kB (UEM) 34*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 650768kB Aug 1 10:30:14 oak-gw06 kernel: Node 0 Normal: 1180*4kB (UEM) 12967*8kB (UEM) 95886*16kB (UEM) 32872*32kB (UEM) 2956*64kB (UEM) 53*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2890504kB Aug 1 10:30:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:30:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:30:14 oak-gw06 kernel: 2098282 total pagecache pages Aug 1 10:30:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:30:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:30:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:30:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:30:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:30:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:30:14 oak-gw06 kernel: 127313 pages reserved Aug 1 10:30:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 10:30:14 oak-gw06 kernel: CPU: 2 PID: 11865 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:30:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:30:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:30:14 oak-gw06 kernel: 00000000000080d0 000000003a207b03 ffff88026f1e7808 ffffffff8168662f Aug 1 10:30:14 oak-gw06 kernel: ffff88026f1e7898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 1 10:30:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88026f1e7868 000000003a207b03 Aug 1 10:30:14 oak-gw06 kernel: Call Trace: Aug 1 10:30:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:30:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:30:14 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 1 10:30:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:30:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:30:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:30:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:30:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 10:30:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 10:30:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:30:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:30:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:30:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:30:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:30:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:30:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:30:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:30:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:30:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:30:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:30:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:30:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:30:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:30:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:30:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:30:14 oak-gw06 kernel: Mem-Info: Aug 1 10:30:14 oak-gw06 kernel: active_anon:29303 inactive_anon:41519 isolated_anon:0#012 active_file:2055865 inactive_file:10816 isolated_file:0#012 unevictable:0 dirty:25872 writeback:9099 unstable:0#012 slab_reclaimable:37113 slab_unreclaimable:804826#012 mapped:5751 shmem:39000 pagetables:1642 bounce:0#012 free:886728 free_pcp:1078 free_cma:0 Aug 1 10:30:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:30:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:30:14 oak-gw06 kernel: Node 0 DMA32 free:650192kB min:11976kB low:14968kB high:17964kB active_anon:16948kB inactive_anon:31248kB active_file:1438812kB inactive_file:9396kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:19336kB writeback:6384kB mapped:1996kB shmem:31276kB slab_reclaimable:27432kB slab_unreclaimable:579272kB kernel_stack:1040kB pagetables:1124kB unstable:0kB bounce:0kB free_pcp:1816kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:30:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:30:14 oak-gw06 kernel: Node 0 Normal free:2873632kB min:55536kB low:69420kB high:83304kB active_anon:100524kB inactive_anon:134828kB active_file:6784648kB inactive_file:47052kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:94488kB writeback:33504kB mapped:21008kB shmem:124724kB slab_reclaimable:121020kB slab_unreclaimable:2640016kB kernel_stack:4640kB pagetables:5444kB unstable:0kB bounce:0kB free_pcp:2008kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:30:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:30:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:30:14 oak-gw06 kernel: Node 0 DMA32: 477*4kB (UE) 614*8kB (UEM) 13599*16kB (UEM) 10456*32kB (UEM) 1328*64kB (UEM) 34*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 648596kB Aug 1 10:30:14 oak-gw06 kernel: Node 0 Normal: 3254*4kB (UEM) 8682*8kB (UEM) 96307*16kB (UEM) 32886*32kB (UEM) 2956*64kB (UEM) 53*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2871704kB Aug 1 10:30:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:30:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:30:14 oak-gw06 kernel: 2108163 total pagecache pages Aug 1 10:30:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:30:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:30:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:30:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:30:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:30:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:30:14 oak-gw06 kernel: 127313 pages reserved Aug 1 10:35:14 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 10:35:14 oak-gw06 kernel: CPU: 2 PID: 11900 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:35:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:35:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:35:14 oak-gw06 kernel: 00000000000080d0 00000000497eabc8 ffff880404027858 ffffffff8168662f Aug 1 10:35:14 oak-gw06 kernel: ffff8804040278e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 10:35:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8804040278b8 00000000497eabc8 Aug 1 10:35:14 oak-gw06 kernel: Call Trace: Aug 1 10:35:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:35:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:35:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:35:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:35:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 10:35:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 10:35:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:35:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:35:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:35:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:35:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:35:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:35:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:35:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:35:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:35:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:35:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:35:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:35:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:35:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:35:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:35:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:35:14 oak-gw06 kernel: Mem-Info: Aug 1 10:35:14 oak-gw06 kernel: active_anon:16006 inactive_anon:41519 isolated_anon:0#012 active_file:2069691 inactive_file:1302 isolated_file:0#012 unevictable:0 dirty:501 writeback:40 unstable:0#012 slab_reclaimable:37477 slab_unreclaimable:802274#012 mapped:5498 shmem:39000 pagetables:1352 bounce:0#012 free:1006206 free_pcp:187 free_cma:0 Aug 1 10:35:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:35:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:35:14 oak-gw06 kernel: Node 0 DMA32 free:871160kB min:11976kB low:14968kB high:17964kB active_anon:13152kB inactive_anon:31248kB active_file:1330496kB inactive_file:536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:48kB writeback:0kB mapped:1876kB shmem:31276kB slab_reclaimable:27660kB slab_unreclaimable:574896kB kernel_stack:1056kB pagetables:1048kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:35:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:35:14 oak-gw06 kernel: Node 0 Normal free:3137772kB min:55536kB low:69420kB high:83304kB active_anon:51132kB inactive_anon:134828kB active_file:6948268kB inactive_file:4672kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:1956kB writeback:160kB mapped:20116kB shmem:124724kB slab_reclaimable:122248kB slab_unreclaimable:2634184kB kernel_stack:4624kB pagetables:4360kB unstable:0kB bounce:0kB free_pcp:544kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:35:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:35:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:35:14 oak-gw06 kernel: Node 0 DMA32: 11565*4kB (UEM) 8143*8kB (UEM) 18196*16kB (UEM) 11247*32kB (UEM) 1602*64kB (UEM) 45*128kB (UM) 2*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 871244kB Aug 1 10:35:14 oak-gw06 kernel: Node 0 Normal: 25081*4kB (UEM) 52286*8kB (UEM) 89988*16kB (UEM) 29960*32kB (UEM) 3227*64kB (UEM) 107*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3137620kB Aug 1 10:35:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:35:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:35:14 oak-gw06 kernel: 2109995 total pagecache pages Aug 1 10:35:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:35:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:35:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:35:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:35:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:35:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:35:14 oak-gw06 kernel: 127313 pages reserved Aug 1 10:35:14 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 10:35:14 oak-gw06 kernel: CPU: 4 PID: 11900 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:35:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:35:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:35:14 oak-gw06 kernel: 00000000000080d0 00000000497eabc8 ffff880404027808 ffffffff8168662f Aug 1 10:35:14 oak-gw06 kernel: ffff880404027898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 1 10:35:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880404027868 00000000497eabc8 Aug 1 10:35:14 oak-gw06 kernel: Call Trace: Aug 1 10:35:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:35:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:35:14 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 1 10:35:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:35:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:35:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:35:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:35:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 10:35:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 10:35:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:35:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:35:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:35:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:35:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:35:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:35:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:35:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:35:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:35:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:35:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:35:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:35:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:35:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:35:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:35:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:35:14 oak-gw06 kernel: Mem-Info: Aug 1 10:35:14 oak-gw06 kernel: active_anon:16006 inactive_anon:41519 isolated_anon:0#012 active_file:2069626 inactive_file:1302 isolated_file:0#012 unevictable:0 dirty:515 writeback:17 unstable:0#012 slab_reclaimable:37477 slab_unreclaimable:802278#012 mapped:5498 shmem:39000 pagetables:1352 bounce:0#012 free:1006384 free_pcp:190 free_cma:0 Aug 1 10:35:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:35:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:35:14 oak-gw06 kernel: Node 0 DMA32 free:871244kB min:11976kB low:14968kB high:17964kB active_anon:13152kB inactive_anon:31248kB active_file:1330496kB inactive_file:536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:48kB writeback:0kB mapped:1876kB shmem:31276kB slab_reclaimable:27660kB slab_unreclaimable:574896kB kernel_stack:1056kB pagetables:1048kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:35:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:35:14 oak-gw06 kernel: Node 0 Normal free:3137656kB min:55536kB low:69420kB high:83304kB active_anon:50872kB inactive_anon:134828kB active_file:6948008kB inactive_file:4672kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:2012kB writeback:68kB mapped:20116kB shmem:124724kB slab_reclaimable:122248kB slab_unreclaimable:2634200kB kernel_stack:4624kB pagetables:4360kB unstable:0kB bounce:0kB free_pcp:1380kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:35:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:35:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:35:14 oak-gw06 kernel: Node 0 DMA32: 11566*4kB (UEM) 8143*8kB (UEM) 18196*16kB (UEM) 11247*32kB (UEM) 1602*64kB (UEM) 45*128kB (UM) 2*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 871248kB Aug 1 10:35:14 oak-gw06 kernel: Node 0 Normal: 25081*4kB (UEM) 52286*8kB (UEM) 89988*16kB (UEM) 29960*32kB (UEM) 3227*64kB (UEM) 107*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3137620kB Aug 1 10:35:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:35:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:35:14 oak-gw06 kernel: 2109898 total pagecache pages Aug 1 10:35:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:35:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:35:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:35:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:35:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:35:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:35:14 oak-gw06 kernel: 127313 pages reserved Aug 1 10:40:14 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 1 10:40:14 oak-gw06 kernel: CPU: 2 PID: 11959 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:40:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:40:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:40:14 oak-gw06 kernel: 00000000000080d0 000000005f3bc3ce ffff88015eb17858 ffffffff8168662f Aug 1 10:40:14 oak-gw06 kernel: ffff88015eb178e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 10:40:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88015eb178b8 000000005f3bc3ce Aug 1 10:40:14 oak-gw06 kernel: Call Trace: Aug 1 10:40:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:40:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:40:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:40:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:40:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 10:40:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 10:40:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:40:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:40:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:40:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:40:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:40:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:40:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:40:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:40:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:40:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:40:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:40:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:40:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:40:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:40:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:40:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:40:14 oak-gw06 kernel: Mem-Info: Aug 1 10:40:14 oak-gw06 kernel: active_anon:17304 inactive_anon:41519 isolated_anon:0#012 active_file:1923040 inactive_file:11963 isolated_file:0#012 unevictable:0 dirty:6222 writeback:0 unstable:0#012 slab_reclaimable:38580 slab_unreclaimable:801587#012 mapped:5688 shmem:39000 pagetables:1564 bounce:0#012 free:1140072 free_pcp:131 free_cma:0 Aug 1 10:40:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:40:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:40:14 oak-gw06 kernel: Node 0 DMA32 free:1234036kB min:11976kB low:14968kB high:17964kB active_anon:13796kB inactive_anon:31248kB active_file:949832kB inactive_file:6916kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2100kB writeback:0kB mapped:1996kB shmem:31276kB slab_reclaimable:27788kB slab_unreclaimable:569476kB kernel_stack:1040kB pagetables:1052kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:40:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:40:14 oak-gw06 kernel: Node 0 Normal free:3310048kB min:55536kB low:69420kB high:83304kB active_anon:55680kB inactive_anon:134828kB active_file:6742328kB inactive_file:40936kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:22788kB writeback:0kB mapped:20756kB shmem:124724kB slab_reclaimable:126532kB slab_unreclaimable:2636856kB kernel_stack:4704kB pagetables:5204kB unstable:0kB bounce:0kB free_pcp:640kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:40:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:40:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:40:14 oak-gw06 kernel: Node 0 DMA32: 16462*4kB (UEM) 33132*8kB (UEM) 24337*16kB (UEM) 11984*32kB (UEM) 1885*64kB (UEM) 73*128kB (UM) 2*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1234280kB Aug 1 10:40:14 oak-gw06 kernel: Node 0 Normal: 87715*4kB (UEM) 73345*8kB (UEM) 82382*16kB (UEM) 26501*32kB (UEM) 2997*64kB (UEM) 110*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3309908kB Aug 1 10:40:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:40:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:40:14 oak-gw06 kernel: 1974005 total pagecache pages Aug 1 10:40:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:40:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:40:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:40:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:40:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:40:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:40:14 oak-gw06 kernel: 127313 pages reserved Aug 1 10:40:14 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 1 10:40:14 oak-gw06 kernel: CPU: 2 PID: 11959 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:40:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:40:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:40:14 oak-gw06 kernel: 00000000000080d0 000000005f3bc3ce ffff88015eb17808 ffffffff8168662f Aug 1 10:40:14 oak-gw06 kernel: ffff88015eb17898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 10:40:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88015eb17868 000000005f3bc3ce Aug 1 10:40:14 oak-gw06 kernel: Call Trace: Aug 1 10:40:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:40:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:40:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:40:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:40:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:40:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:40:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 10:40:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 10:40:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:40:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:40:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:40:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:40:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:40:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:40:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:40:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:40:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:40:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:40:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:40:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:40:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:40:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:40:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:40:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:40:14 oak-gw06 kernel: Mem-Info: Aug 1 10:40:14 oak-gw06 kernel: active_anon:17369 inactive_anon:41519 isolated_anon:0#012 active_file:1923040 inactive_file:11963 isolated_file:0#012 unevictable:0 dirty:6222 writeback:0 unstable:0#012 slab_reclaimable:38580 slab_unreclaimable:801587#012 mapped:5688 shmem:39000 pagetables:1564 bounce:0#012 free:1140096 free_pcp:116 free_cma:0 Aug 1 10:40:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:40:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:40:14 oak-gw06 kernel: Node 0 DMA32 free:1234036kB min:11976kB low:14968kB high:17964kB active_anon:13796kB inactive_anon:31248kB active_file:949832kB inactive_file:6916kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2100kB writeback:0kB mapped:1996kB shmem:31276kB slab_reclaimable:27788kB slab_unreclaimable:569476kB kernel_stack:1040kB pagetables:1052kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:40:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:40:14 oak-gw06 kernel: Node 0 Normal free:3309460kB min:55536kB low:69420kB high:83304kB active_anon:56200kB inactive_anon:134828kB active_file:6742328kB inactive_file:40936kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:22788kB writeback:0kB mapped:20756kB shmem:124724kB slab_reclaimable:126532kB slab_unreclaimable:2636856kB kernel_stack:4704kB pagetables:5204kB unstable:0kB bounce:0kB free_pcp:712kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:40:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:40:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:40:14 oak-gw06 kernel: Node 0 DMA32: 16465*4kB (UEM) 33147*8kB (UEM) 24343*16kB (UEM) 11985*32kB (UEM) 1885*64kB (UEM) 73*128kB (UM) 2*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1234540kB Aug 1 10:40:14 oak-gw06 kernel: Node 0 Normal: 87680*4kB (UEM) 73377*8kB (UEM) 82380*16kB (UEM) 26502*32kB (UEM) 2997*64kB (UEM) 110*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3310024kB Aug 1 10:40:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:40:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:40:14 oak-gw06 kernel: 1974005 total pagecache pages Aug 1 10:40:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:40:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:40:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:40:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:40:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:40:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:40:14 oak-gw06 kernel: 127313 pages reserved Aug 1 10:45:14 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 10:45:14 oak-gw06 kernel: CPU: 6 PID: 12049 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:45:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:45:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:45:14 oak-gw06 kernel: 00000000000080d0 00000000b721f32c ffff8801ffcef858 ffffffff8168662f Aug 1 10:45:14 oak-gw06 kernel: ffff8801ffcef8e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 1 10:45:14 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff8801ffcef8e8 00000000b721f32c Aug 1 10:45:14 oak-gw06 kernel: Call Trace: Aug 1 10:45:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:45:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:45:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:45:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:45:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 10:45:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 10:45:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:45:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:45:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:45:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:45:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:45:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:45:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:45:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:45:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:45:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:45:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:45:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:45:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:45:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:45:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:45:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:45:14 oak-gw06 kernel: Mem-Info: Aug 1 10:45:14 oak-gw06 kernel: active_anon:28123 inactive_anon:41519 isolated_anon:0#012 active_file:2066892 inactive_file:2406 isolated_file:0#012 unevictable:0 dirty:43334 writeback:3934 unstable:0#012 slab_reclaimable:38791 slab_unreclaimable:806429#012 mapped:5787 shmem:39000 pagetables:1643 bounce:0#012 free:882117 free_pcp:879 free_cma:0 Aug 1 10:45:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:45:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:45:14 oak-gw06 kernel: Node 0 DMA32 free:934976kB min:11976kB low:14968kB high:17964kB active_anon:17004kB inactive_anon:31248kB active_file:1180020kB inactive_file:468kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:33004kB writeback:20kB mapped:2000kB shmem:31276kB slab_reclaimable:27832kB slab_unreclaimable:572028kB kernel_stack:1040kB pagetables:1176kB unstable:0kB bounce:0kB free_pcp:628kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:45:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:45:14 oak-gw06 kernel: Node 0 Normal free:2577852kB min:55536kB low:69420kB high:83304kB active_anon:95488kB inactive_anon:134828kB active_file:7087548kB inactive_file:13316kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:140332kB writeback:19596kB mapped:21148kB shmem:124724kB slab_reclaimable:127332kB slab_unreclaimable:2653672kB kernel_stack:4640kB pagetables:5396kB unstable:0kB bounce:0kB free_pcp:2964kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:45:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:45:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:45:14 oak-gw06 kernel: Node 0 DMA32: 1072*4kB (UEM) 2686*8kB (UEM) 19137*16kB (UEM) 13136*32kB (UEM) 2545*64kB (UEM) 145*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 934016kB Aug 1 10:45:14 oak-gw06 kernel: Node 0 Normal: 3393*4kB (UEM) 76228*8kB (UEM) 64861*16kB (UEM) 21631*32kB (UEM) 3196*64kB (UEM) 148*128kB (UEM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2577364kB Aug 1 10:45:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:45:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:45:14 oak-gw06 kernel: 2110027 total pagecache pages Aug 1 10:45:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:45:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:45:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:45:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:45:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:45:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:45:14 oak-gw06 kernel: 127313 pages reserved Aug 1 10:45:14 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 10:45:14 oak-gw06 kernel: CPU: 6 PID: 12049 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:45:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:45:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:45:14 oak-gw06 kernel: 00000000000080d0 00000000b721f32c ffff8801ffcef808 ffffffff8168662f Aug 1 10:45:14 oak-gw06 kernel: ffff8801ffcef898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 1 10:45:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801ffcef868 00000000b721f32c Aug 1 10:45:14 oak-gw06 kernel: Call Trace: Aug 1 10:45:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:45:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:45:14 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 1 10:45:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:45:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:45:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:45:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:45:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 10:45:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 10:45:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:45:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:45:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:45:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:45:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:45:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:45:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:45:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:45:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:45:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:45:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:45:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:45:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:45:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:45:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:45:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:45:14 oak-gw06 kernel: Mem-Info: Aug 1 10:45:14 oak-gw06 kernel: active_anon:27928 inactive_anon:41519 isolated_anon:0#012 active_file:2070222 inactive_file:671 isolated_file:0#012 unevictable:0 dirty:43189 writeback:5681 unstable:0#012 slab_reclaimable:38791 slab_unreclaimable:806429#012 mapped:5787 shmem:39000 pagetables:1643 bounce:0#012 free:881924 free_pcp:507 free_cma:0 Aug 1 10:45:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:45:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:45:14 oak-gw06 kernel: Node 0 DMA32 free:932684kB min:11976kB low:14968kB high:17964kB active_anon:17004kB inactive_anon:31248kB active_file:1181900kB inactive_file:548kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:33588kB writeback:1188kB mapped:2000kB shmem:31276kB slab_reclaimable:27832kB slab_unreclaimable:572028kB kernel_stack:1040kB pagetables:1176kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:45:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:45:14 oak-gw06 kernel: Node 0 Normal free:2577456kB min:55536kB low:69420kB high:83304kB active_anon:94968kB inactive_anon:134828kB active_file:7098988kB inactive_file:2136kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:139168kB writeback:21536kB mapped:21148kB shmem:124724kB slab_reclaimable:127332kB slab_unreclaimable:2653672kB kernel_stack:4640kB pagetables:5396kB unstable:0kB bounce:0kB free_pcp:2076kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:45:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:45:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:45:14 oak-gw06 kernel: Node 0 DMA32: 1039*4kB (UEM) 2687*8kB (UEM) 19122*16kB (UEM) 13136*32kB (UEM) 2545*64kB (UEM) 145*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 933652kB Aug 1 10:45:14 oak-gw06 kernel: Node 0 Normal: 3545*4kB (UEM) 76479*8kB (UEM) 64723*16kB (UEM) 21615*32kB (UEM) 3195*64kB (UEM) 148*128kB (UEM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2577196kB Aug 1 10:45:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:45:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:45:14 oak-gw06 kernel: 2109930 total pagecache pages Aug 1 10:45:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:45:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:45:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:45:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:45:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:45:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:45:14 oak-gw06 kernel: 127313 pages reserved Aug 1 10:50:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 10:50:14 oak-gw06 kernel: CPU: 2 PID: 12075 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:50:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:50:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:50:14 oak-gw06 kernel: 00000000000080d0 00000000cccc9fe3 ffff8800b2bb7858 ffffffff8168662f Aug 1 10:50:14 oak-gw06 kernel: ffff8800b2bb78e8 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 1 10:50:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800b2bb78b8 00000000cccc9fe3 Aug 1 10:50:14 oak-gw06 kernel: Call Trace: Aug 1 10:50:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:50:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:50:14 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 1 10:50:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:50:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:50:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 10:50:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 10:50:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:50:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:50:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:50:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:50:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:50:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:50:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:50:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:50:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:50:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:50:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:50:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:50:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:50:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:50:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:50:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:50:14 oak-gw06 kernel: Mem-Info: Aug 1 10:50:14 oak-gw06 kernel: active_anon:17945 inactive_anon:41519 isolated_anon:0#012 active_file:1972420 inactive_file:99208 isolated_file:0#012 unevictable:0 dirty:2312 writeback:256 unstable:0#012 slab_reclaimable:37657 slab_unreclaimable:804884#012 mapped:5800 shmem:39000 pagetables:1453 bounce:0#012 free:1000737 free_pcp:65 free_cma:0 Aug 1 10:50:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:50:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:50:14 oak-gw06 kernel: Node 0 DMA32 free:952892kB min:11976kB low:14968kB high:17964kB active_anon:13152kB inactive_anon:31248kB active_file:1175940kB inactive_file:83584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4kB writeback:0kB mapped:2044kB shmem:31276kB slab_reclaimable:27412kB slab_unreclaimable:571128kB kernel_stack:1056kB pagetables:1048kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:50:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:50:14 oak-gw06 kernel: Node 0 Normal free:3033420kB min:55536kB low:69420kB high:83304kB active_anon:59148kB inactive_anon:134828kB active_file:6713740kB inactive_file:313248kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9244kB writeback:1024kB mapped:21156kB shmem:124724kB slab_reclaimable:123216kB slab_unreclaimable:2648392kB kernel_stack:4640kB pagetables:4764kB unstable:0kB bounce:0kB free_pcp:376kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:50:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:50:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:50:14 oak-gw06 kernel: Node 0 DMA32: 10077*4kB (UEM) 5823*8kB (UEM) 15030*16kB (UEM) 13324*32kB (UEM) 2772*64kB (UEM) 168*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 952908kB Aug 1 10:50:14 oak-gw06 kernel: Node 0 Normal: 24669*4kB (UEM) 86752*8kB (UEM) 71375*16kB (UEM) 25605*32kB (UEM) 3855*64kB (UEM) 253*128kB (UEM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3034180kB Aug 1 10:50:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:50:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:50:14 oak-gw06 kernel: 2110628 total pagecache pages Aug 1 10:50:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:50:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:50:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:50:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:50:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:50:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:50:14 oak-gw06 kernel: 127313 pages reserved Aug 1 10:50:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 10:50:14 oak-gw06 kernel: CPU: 2 PID: 12075 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:50:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:50:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:50:14 oak-gw06 kernel: 00000000000080d0 00000000cccc9fe3 ffff8800b2bb7808 ffffffff8168662f Aug 1 10:50:14 oak-gw06 kernel: ffff8800b2bb7898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 10:50:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800b2bb7868 00000000cccc9fe3 Aug 1 10:50:14 oak-gw06 kernel: Call Trace: Aug 1 10:50:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:50:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:50:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:50:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:50:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:50:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:50:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 10:50:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 10:50:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:50:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:50:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:50:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:50:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:50:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:50:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:50:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:50:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:50:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:50:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:50:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:50:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:50:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:50:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:50:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:50:14 oak-gw06 kernel: Mem-Info: Aug 1 10:50:14 oak-gw06 kernel: active_anon:17945 inactive_anon:41519 isolated_anon:0#012 active_file:1972355 inactive_file:99208 isolated_file:0#012 unevictable:0 dirty:2312 writeback:256 unstable:0#012 slab_reclaimable:37657 slab_unreclaimable:804884#012 mapped:5800 shmem:39000 pagetables:1453 bounce:0#012 free:1000811 free_pcp:31 free_cma:0 Aug 1 10:50:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:50:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:50:14 oak-gw06 kernel: Node 0 DMA32 free:952892kB min:11976kB low:14968kB high:17964kB active_anon:13152kB inactive_anon:31248kB active_file:1175940kB inactive_file:83584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4kB writeback:0kB mapped:2044kB shmem:31276kB slab_reclaimable:27412kB slab_unreclaimable:571128kB kernel_stack:1056kB pagetables:1048kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:50:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:50:14 oak-gw06 kernel: Node 0 Normal free:3034460kB min:55536kB low:69420kB high:83304kB active_anon:58628kB inactive_anon:134828kB active_file:6713480kB inactive_file:313248kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9244kB writeback:1024kB mapped:21156kB shmem:124724kB slab_reclaimable:123216kB slab_unreclaimable:2648392kB kernel_stack:4640kB pagetables:4764kB unstable:0kB bounce:0kB free_pcp:216kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:50:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:50:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:50:14 oak-gw06 kernel: Node 0 DMA32: 10077*4kB (UEM) 5823*8kB (UEM) 15030*16kB (UEM) 13324*32kB (UEM) 2772*64kB (UEM) 168*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 952908kB Aug 1 10:50:14 oak-gw06 kernel: Node 0 Normal: 24790*4kB (UEM) 86755*8kB (UEM) 71376*16kB (UEM) 25605*32kB (UEM) 3855*64kB (UEM) 253*128kB (UEM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3034704kB Aug 1 10:50:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:50:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:50:14 oak-gw06 kernel: 2110531 total pagecache pages Aug 1 10:50:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:50:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:50:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:50:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:50:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:50:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:50:14 oak-gw06 kernel: 127313 pages reserved Aug 1 10:55:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 10:55:14 oak-gw06 kernel: CPU: 2 PID: 12075 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:55:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:55:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:55:14 oak-gw06 kernel: 00000000000080d0 00000000cccc9fe3 ffff8800b2bb7858 ffffffff8168662f Aug 1 10:55:14 oak-gw06 kernel: ffff8800b2bb78e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 1 10:55:15 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff8800b2bb78e8 00000000cccc9fe3 Aug 1 10:55:15 oak-gw06 kernel: Call Trace: Aug 1 10:55:15 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:55:15 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:55:15 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:55:15 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:55:15 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 10:55:15 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 10:55:15 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:55:15 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:55:15 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:55:15 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:55:15 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:55:15 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:55:15 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:55:15 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:55:15 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:55:15 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:55:15 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:55:15 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:55:15 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:55:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:55:15 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:55:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:55:15 oak-gw06 kernel: Mem-Info: Aug 1 10:55:15 oak-gw06 kernel: active_anon:22680 inactive_anon:41519 isolated_anon:0#012 active_file:1947947 inactive_file:63735 isolated_file:0#012 unevictable:0 dirty:2812 writeback:388 unstable:0#012 slab_reclaimable:38412 slab_unreclaimable:805180#012 mapped:5811 shmem:39000 pagetables:1619 bounce:0#012 free:1053048 free_pcp:723 free_cma:0 Aug 1 10:55:15 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:55:15 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:55:15 oak-gw06 kernel: Node 0 DMA32 free:1271732kB min:11976kB low:14968kB high:17964kB active_anon:13152kB inactive_anon:31248kB active_file:890344kB inactive_file:58128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:8kB writeback:0kB mapped:2080kB shmem:31276kB slab_reclaimable:27396kB slab_unreclaimable:563224kB kernel_stack:1024kB pagetables:1048kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:55:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:55:15 oak-gw06 kernel: Node 0 Normal free:2923912kB min:55536kB low:69420kB high:83304kB active_anon:78348kB inactive_anon:134828kB active_file:6901444kB inactive_file:196812kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:11240kB writeback:1552kB mapped:21164kB shmem:124724kB slab_reclaimable:126252kB slab_unreclaimable:2657480kB kernel_stack:4656kB pagetables:5428kB unstable:0kB bounce:0kB free_pcp:2820kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:55:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:55:15 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:55:15 oak-gw06 kernel: Node 0 DMA32: 31799*4kB (UEM) 24302*8kB (UEM) 19455*16kB (UEM) 13527*32kB (UEM) 2856*64kB (UEM) 183*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1272220kB Aug 1 10:55:15 oak-gw06 kernel: Node 0 Normal: 32015*4kB (UEM) 91956*8kB (UEM) 63461*16kB (UEM) 23997*32kB (UEM) 3806*64kB (UEM) 257*128kB (UEM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2924492kB Aug 1 10:55:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:55:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:55:15 oak-gw06 kernel: 2050696 total pagecache pages Aug 1 10:55:15 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:55:15 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:55:15 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:55:15 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:55:15 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:55:15 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:55:15 oak-gw06 kernel: 127313 pages reserved Aug 1 10:55:15 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 10:55:15 oak-gw06 kernel: CPU: 5 PID: 12075 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 10:55:15 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 10:55:15 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 10:55:15 oak-gw06 kernel: 00000000000080d0 00000000cccc9fe3 ffff8800b2bb7808 ffffffff8168662f Aug 1 10:55:15 oak-gw06 kernel: ffff8800b2bb7898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 10:55:15 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800b2bb7868 00000000cccc9fe3 Aug 1 10:55:15 oak-gw06 kernel: Call Trace: Aug 1 10:55:15 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 10:55:15 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 10:55:15 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 10:55:15 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 10:55:15 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 10:55:15 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 10:55:15 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 10:55:15 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 10:55:15 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 10:55:15 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 10:55:15 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 10:55:15 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 10:55:15 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 10:55:15 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 10:55:15 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 10:55:15 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 10:55:15 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 10:55:15 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 10:55:15 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 10:55:15 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 10:55:15 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 10:55:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:55:15 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 10:55:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 10:55:15 oak-gw06 kernel: Mem-Info: Aug 1 10:55:15 oak-gw06 kernel: active_anon:22745 inactive_anon:41519 isolated_anon:0#012 active_file:1947817 inactive_file:63735 isolated_file:0#012 unevictable:0 dirty:2812 writeback:291 unstable:0#012 slab_reclaimable:38412 slab_unreclaimable:805180#012 mapped:5811 shmem:39000 pagetables:1619 bounce:0#012 free:1053599 free_pcp:189 free_cma:0 Aug 1 10:55:15 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 10:55:15 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 10:55:15 oak-gw06 kernel: Node 0 DMA32 free:1271884kB min:11976kB low:14968kB high:17964kB active_anon:13152kB inactive_anon:31248kB active_file:890344kB inactive_file:58128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:8kB writeback:0kB mapped:2080kB shmem:31276kB slab_reclaimable:27396kB slab_unreclaimable:563224kB kernel_stack:1024kB pagetables:1048kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:55:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 10:55:15 oak-gw06 kernel: Node 0 Normal free:2926144kB min:55536kB low:69420kB high:83304kB active_anon:78380kB inactive_anon:134828kB active_file:6900980kB inactive_file:196816kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:11164kB writeback:1124kB mapped:21164kB shmem:124724kB slab_reclaimable:126252kB slab_unreclaimable:2657644kB kernel_stack:4656kB pagetables:5420kB unstable:0kB bounce:0kB free_pcp:864kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 10:55:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 10:55:15 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 10:55:15 oak-gw06 kernel: Node 0 DMA32: 31799*4kB (UEM) 24302*8kB (UEM) 19455*16kB (UEM) 13527*32kB (UEM) 2856*64kB (UEM) 183*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1272220kB Aug 1 10:55:15 oak-gw06 kernel: Node 0 Normal: 32534*4kB (UEM) 91968*8kB (UEM) 63465*16kB (UEM) 23997*32kB (UEM) 3806*64kB (UEM) 257*128kB (UEM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2926728kB Aug 1 10:55:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 10:55:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 10:55:15 oak-gw06 kernel: 2050521 total pagecache pages Aug 1 10:55:15 oak-gw06 kernel: 0 pages in swap cache Aug 1 10:55:15 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 10:55:15 oak-gw06 kernel: Free swap = 4194300kB Aug 1 10:55:15 oak-gw06 kernel: Total swap = 4194300kB Aug 1 10:55:15 oak-gw06 kernel: 4194203 pages RAM Aug 1 10:55:15 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 10:55:15 oak-gw06 kernel: 127313 pages reserved Aug 1 11:00:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 11:00:14 oak-gw06 kernel: CPU: 2 PID: 12075 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:00:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:00:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:00:14 oak-gw06 kernel: 00000000000080d0 00000000cccc9fe3 ffff8800b2bb7858 ffffffff8168662f Aug 1 11:00:14 oak-gw06 kernel: ffff8800b2bb78e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:00:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800b2bb78b8 00000000cccc9fe3 Aug 1 11:00:14 oak-gw06 kernel: Call Trace: Aug 1 11:00:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:00:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:00:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:00:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:00:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 11:00:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 11:00:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:00:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:00:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:00:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:00:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:00:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:00:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:00:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:00:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:00:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:00:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:00:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:00:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:00:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:00:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:00:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:00:14 oak-gw06 kernel: Mem-Info: Aug 1 11:00:14 oak-gw06 kernel: active_anon:21896 inactive_anon:41519 isolated_anon:0#012 active_file:2052399 inactive_file:21010 isolated_file:0#012 unevictable:0 dirty:2732 writeback:3 unstable:0#012 slab_reclaimable:39719 slab_unreclaimable:810657#012 mapped:5822 shmem:39000 pagetables:1604 bounce:0#012 free:986400 free_pcp:123 free_cma:0 Aug 1 11:00:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:00:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:00:14 oak-gw06 kernel: Node 0 DMA32 free:1101088kB min:11976kB low:14968kB high:17964kB active_anon:13152kB inactive_anon:31248kB active_file:1087608kB inactive_file:13848kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:808kB writeback:0kB mapped:2120kB shmem:31276kB slab_reclaimable:27916kB slab_unreclaimable:567176kB kernel_stack:1024kB pagetables:1048kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:00:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:00:14 oak-gw06 kernel: Node 0 Normal free:2828348kB min:55536kB low:69420kB high:83304kB active_anon:74432kB inactive_anon:134828kB active_file:7121988kB inactive_file:70192kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10120kB writeback:12kB mapped:21168kB shmem:124724kB slab_reclaimable:130960kB slab_unreclaimable:2675436kB kernel_stack:4704kB pagetables:5368kB unstable:0kB bounce:0kB free_pcp:624kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:00:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:00:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:00:14 oak-gw06 kernel: Node 0 DMA32: 5180*4kB (UEM) 3379*8kB (UEM) 21507*16kB (UEM) 14474*32kB (UEM) 3350*64kB (UEM) 254*128kB (UEM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1102456kB Aug 1 11:00:14 oak-gw06 kernel: Node 0 Normal: 19610*4kB (UEM) 91130*8kB (UEM) 61527*16kB (UEM) 23550*32kB (UEM) 3868*64kB (UEM) 266*128kB (UEM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2828136kB Aug 1 11:00:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:00:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:00:14 oak-gw06 kernel: 2112415 total pagecache pages Aug 1 11:00:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:00:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:00:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:00:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:00:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:00:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:00:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:00:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 11:00:14 oak-gw06 kernel: CPU: 2 PID: 12075 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:00:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:00:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:00:14 oak-gw06 kernel: 00000000000080d0 00000000cccc9fe3 ffff8800b2bb7808 ffffffff8168662f Aug 1 11:00:14 oak-gw06 kernel: ffff8800b2bb7898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:00:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800b2bb7868 00000000cccc9fe3 Aug 1 11:00:14 oak-gw06 kernel: Call Trace: Aug 1 11:00:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:00:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:00:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:00:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:00:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 11:00:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 11:00:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 11:00:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 11:00:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:00:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:00:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:00:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:00:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:00:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:00:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:00:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:00:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:00:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:00:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:00:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:00:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:00:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:00:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:00:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:00:14 oak-gw06 kernel: Mem-Info: Aug 1 11:00:14 oak-gw06 kernel: active_anon:21896 inactive_anon:41519 isolated_anon:0#012 active_file:2052334 inactive_file:21010 isolated_file:0#012 unevictable:0 dirty:2732 writeback:3 unstable:0#012 slab_reclaimable:39719 slab_unreclaimable:810794#012 mapped:5822 shmem:39000 pagetables:1604 bounce:0#012 free:986590 free_pcp:137 free_cma:0 Aug 1 11:00:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:00:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:00:14 oak-gw06 kernel: Node 0 DMA32 free:1101744kB min:11976kB low:14968kB high:17964kB active_anon:13152kB inactive_anon:31248kB active_file:1087608kB inactive_file:13848kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:808kB writeback:0kB mapped:2120kB shmem:31276kB slab_reclaimable:27916kB slab_unreclaimable:567176kB kernel_stack:1024kB pagetables:1048kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:00:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:00:14 oak-gw06 kernel: Node 0 Normal free:2828072kB min:55536kB low:69420kB high:83304kB active_anon:74432kB inactive_anon:134828kB active_file:7121728kB inactive_file:70192kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10120kB writeback:12kB mapped:21168kB shmem:124724kB slab_reclaimable:130960kB slab_unreclaimable:2675984kB kernel_stack:4704kB pagetables:5368kB unstable:0kB bounce:0kB free_pcp:836kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:00:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:00:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:00:14 oak-gw06 kernel: Node 0 DMA32: 5180*4kB (UEM) 3379*8kB (UEM) 21515*16kB (UEM) 14474*32kB (UEM) 3350*64kB (UEM) 254*128kB (UEM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1102584kB Aug 1 11:00:14 oak-gw06 kernel: Node 0 Normal: 19602*4kB (UEM) 91157*8kB (UEM) 61515*16kB (UEM) 23547*32kB (UEM) 3868*64kB (UEM) 266*128kB (UEM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2828032kB Aug 1 11:00:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:00:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:00:14 oak-gw06 kernel: 2112318 total pagecache pages Aug 1 11:00:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:00:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:00:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:00:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:00:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:00:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:00:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:05:14 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 1 11:05:14 oak-gw06 kernel: CPU: 2 PID: 12286 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:05:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:05:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:05:14 oak-gw06 kernel: 00000000000080d0 00000000cbe4b452 ffff880233fb3858 ffffffff8168662f Aug 1 11:05:14 oak-gw06 kernel: ffff880233fb38e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:05:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880233fb38b8 00000000cbe4b452 Aug 1 11:05:14 oak-gw06 kernel: Call Trace: Aug 1 11:05:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:05:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:05:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:05:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:05:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 11:05:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 11:05:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:05:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:05:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:05:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:05:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:05:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:05:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:05:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:05:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:05:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:05:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:05:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:05:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:05:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:05:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:05:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:05:14 oak-gw06 kernel: Mem-Info: Aug 1 11:05:14 oak-gw06 kernel: active_anon:21874 inactive_anon:41519 isolated_anon:0#012 active_file:1932914 inactive_file:11437 isolated_file:0#012 unevictable:0 dirty:259 writeback:17 unstable:0#012 slab_reclaimable:39635 slab_unreclaimable:815618#012 mapped:5835 shmem:39000 pagetables:1603 bounce:0#012 free:1110675 free_pcp:116 free_cma:0 Aug 1 11:05:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:05:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:05:14 oak-gw06 kernel: Node 0 DMA32 free:979268kB min:11976kB low:14968kB high:17964kB active_anon:14816kB inactive_anon:31248kB active_file:1213676kB inactive_file:7820kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:20kB writeback:8kB mapped:2168kB shmem:31276kB slab_reclaimable:27856kB slab_unreclaimable:569136kB kernel_stack:1104kB pagetables:1544kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:05:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:05:14 oak-gw06 kernel: Node 0 Normal free:3446552kB min:55536kB low:69420kB high:83304kB active_anon:72680kB inactive_anon:134828kB active_file:6517980kB inactive_file:37928kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:1016kB writeback:60kB mapped:21172kB shmem:124724kB slab_reclaimable:130684kB slab_unreclaimable:2693320kB kernel_stack:4640kB pagetables:4868kB unstable:0kB bounce:0kB free_pcp:1072kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:05:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:05:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:05:14 oak-gw06 kernel: Node 0 DMA32: 14017*4kB (UEM) 8262*8kB (UEM) 9496*16kB (UEM) 14410*32kB (UEM) 3329*64kB (UEM) 251*128kB (UEM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 980916kB Aug 1 11:05:14 oak-gw06 kernel: Node 0 Normal: 95592*4kB (UEM) 119763*8kB (UEM) 63998*16kB (UEM) 24440*32kB (UEM) 4088*64kB (UEM) 293*128kB (UEM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3446680kB Aug 1 11:05:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:05:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:05:14 oak-gw06 kernel: 1983319 total pagecache pages Aug 1 11:05:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:05:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:05:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:05:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:05:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:05:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:05:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:05:15 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 1 11:05:15 oak-gw06 kernel: CPU: 2 PID: 12286 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:05:15 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:05:15 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:05:15 oak-gw06 kernel: 00000000000080d0 00000000cbe4b452 ffff880233fb3808 ffffffff8168662f Aug 1 11:05:15 oak-gw06 kernel: ffff880233fb3898 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 1 11:05:15 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff880233fb3898 00000000cbe4b452 Aug 1 11:05:15 oak-gw06 kernel: Call Trace: Aug 1 11:05:15 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:05:15 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:05:15 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:05:15 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:05:15 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 11:05:15 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 11:05:15 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 11:05:15 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 11:05:15 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:05:15 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:05:15 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:05:15 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:05:15 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:05:15 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:05:15 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:05:15 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:05:15 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:05:15 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:05:15 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:05:15 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:05:15 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:05:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:05:15 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:05:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:05:15 oak-gw06 kernel: Mem-Info: Aug 1 11:05:15 oak-gw06 kernel: active_anon:22938 inactive_anon:41519 isolated_anon:0#012 active_file:1932740 inactive_file:11443 isolated_file:7#012 unevictable:0 dirty:414 writeback:0 unstable:0#012 slab_reclaimable:39635 slab_unreclaimable:815630#012 mapped:5840 shmem:39000 pagetables:1609 bounce:0#012 free:1110021 free_pcp:273 free_cma:0 Aug 1 11:05:15 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:05:15 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:05:15 oak-gw06 kernel: Node 0 DMA32 free:981232kB min:11976kB low:14968kB high:17964kB active_anon:14824kB inactive_anon:31248kB active_file:1213804kB inactive_file:7828kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:172kB writeback:0kB mapped:2184kB shmem:31276kB slab_reclaimable:27856kB slab_unreclaimable:569268kB kernel_stack:1104kB pagetables:1544kB unstable:0kB bounce:0kB free_pcp:172kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:05:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:05:15 oak-gw06 kernel: Node 0 Normal free:3442620kB min:55536kB low:69420kB high:83304kB active_anon:77188kB inactive_anon:134828kB active_file:6517156kB inactive_file:37944kB unevictable:0kB isolated(anon):0kB isolated(file):28kB present:13631488kB managed:13367060kB mlocked:0kB dirty:1484kB writeback:0kB mapped:21176kB shmem:124724kB slab_reclaimable:130684kB slab_unreclaimable:2693236kB kernel_stack:4640kB pagetables:4892kB unstable:0kB bounce:0kB free_pcp:844kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:05:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:05:15 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:05:15 oak-gw06 kernel: Node 0 DMA32: 14904*4kB (UEM) 8545*8kB (UEM) 9408*16kB (UEM) 14319*32kB (UEM) 3313*64kB (UEM) 250*128kB (UEM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 981256kB Aug 1 11:05:15 oak-gw06 kernel: Node 0 Normal: 102340*4kB (UEM) 118698*8kB (UEM) 62842*16kB (UEM) 24301*32kB (UEM) 4088*64kB (UEM) 294*128kB (UEM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3442336kB Aug 1 11:05:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:05:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:05:15 oak-gw06 kernel: 1983199 total pagecache pages Aug 1 11:05:15 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:05:15 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:05:15 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:05:15 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:05:15 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:05:15 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:05:15 oak-gw06 kernel: 127313 pages reserved Aug 1 11:10:14 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 11:10:14 oak-gw06 kernel: CPU: 2 PID: 12466 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:10:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:10:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:10:14 oak-gw06 kernel: 00000000000080d0 000000005510278c ffff8801ecbcf858 ffffffff8168662f Aug 1 11:10:14 oak-gw06 kernel: ffff8801ecbcf8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:10:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801ecbcf8b8 000000005510278c Aug 1 11:10:14 oak-gw06 kernel: Call Trace: Aug 1 11:10:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:10:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:10:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:10:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:10:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 11:10:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 11:10:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:10:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:10:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:10:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:10:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:10:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:10:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:10:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:10:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:10:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:10:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:10:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:10:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:10:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:10:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:10:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:10:14 oak-gw06 kernel: Mem-Info: Aug 1 11:10:14 oak-gw06 kernel: active_anon:22080 inactive_anon:41519 isolated_anon:0#012 active_file:1914390 inactive_file:163515 isolated_file:0#012 unevictable:0 dirty:2443 writeback:370 unstable:0#012 slab_reclaimable:39509 slab_unreclaimable:822598#012 mapped:5851 shmem:39000 pagetables:1605 bounce:0#012 free:969100 free_pcp:95 free_cma:0 Aug 1 11:10:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:10:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:10:14 oak-gw06 kernel: Node 0 DMA32 free:1061968kB min:11976kB low:14968kB high:17964kB active_anon:13152kB inactive_anon:31248kB active_file:1035724kB inactive_file:116288kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:12kB writeback:0kB mapped:2228kB shmem:31276kB slab_reclaimable:27824kB slab_unreclaimable:569516kB kernel_stack:1040kB pagetables:1048kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:10:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:10:14 oak-gw06 kernel: Node 0 Normal free:2801740kB min:55536kB low:69420kB high:83304kB active_anon:72048kB inactive_anon:134828kB active_file:6621836kB inactive_file:538552kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9372kB writeback:2256kB mapped:21176kB shmem:124724kB slab_reclaimable:130212kB slab_unreclaimable:2720860kB kernel_stack:4736kB pagetables:5372kB unstable:0kB bounce:0kB free_pcp:1120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:10:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:10:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:10:14 oak-gw06 kernel: Node 0 DMA32: 9010*4kB (UEM) 7031*8kB (UEM) 16159*16kB (UEM) 14413*32kB (UEM) 3369*64kB (UEM) 264*128kB (UEM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1061968kB Aug 1 11:10:14 oak-gw06 kernel: Node 0 Normal: 17335*4kB (UEM) 93394*8kB (UEM) 60754*16kB (UEM) 22030*32kB (UEM) 4183*64kB (UEM) 309*128kB (UEM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2801804kB Aug 1 11:10:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:10:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:10:14 oak-gw06 kernel: 2117162 total pagecache pages Aug 1 11:10:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:10:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:10:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:10:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:10:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:10:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:10:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:10:14 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 11:10:14 oak-gw06 kernel: CPU: 2 PID: 12466 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:10:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:10:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:10:14 oak-gw06 kernel: 00000000000080d0 000000005510278c ffff8801ecbcf808 ffffffff8168662f Aug 1 11:10:14 oak-gw06 kernel: ffff8801ecbcf898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:10:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801ecbcf868 000000005510278c Aug 1 11:10:14 oak-gw06 kernel: Call Trace: Aug 1 11:10:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:10:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:10:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:10:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:10:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 11:10:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 11:10:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 11:10:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 11:10:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:10:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:10:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:10:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:10:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:10:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:10:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:10:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:10:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:10:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:10:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:10:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:10:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:10:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:10:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:10:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:10:14 oak-gw06 kernel: Mem-Info: Aug 1 11:10:14 oak-gw06 kernel: active_anon:21300 inactive_anon:41519 isolated_anon:0#012 active_file:1914585 inactive_file:163515 isolated_file:0#012 unevictable:0 dirty:2346 writeback:564 unstable:0#012 slab_reclaimable:39509 slab_unreclaimable:822598#012 mapped:5851 shmem:39000 pagetables:1605 bounce:0#012 free:970225 free_pcp:62 free_cma:0 Aug 1 11:10:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:10:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:10:14 oak-gw06 kernel: Node 0 DMA32 free:1061968kB min:11976kB low:14968kB high:17964kB active_anon:13152kB inactive_anon:31248kB active_file:1035724kB inactive_file:116288kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:12kB writeback:0kB mapped:2228kB shmem:31276kB slab_reclaimable:27824kB slab_unreclaimable:569516kB kernel_stack:1040kB pagetables:1048kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:10:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:10:14 oak-gw06 kernel: Node 0 Normal free:2802348kB min:55536kB low:69420kB high:83304kB active_anon:72048kB inactive_anon:134828kB active_file:6622616kB inactive_file:537772kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9372kB writeback:2256kB mapped:21176kB shmem:124724kB slab_reclaimable:130212kB slab_unreclaimable:2720860kB kernel_stack:4736kB pagetables:5372kB unstable:0kB bounce:0kB free_pcp:552kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:10:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:10:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:10:14 oak-gw06 kernel: Node 0 DMA32: 9013*4kB (UEM) 7032*8kB (UEM) 16159*16kB (UEM) 14413*32kB (UEM) 3369*64kB (UEM) 264*128kB (UEM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1061988kB Aug 1 11:10:14 oak-gw06 kernel: Node 0 Normal: 16447*4kB (UEM) 93417*8kB (UEM) 60697*16kB (UEM) 22028*32kB (UEM) 4183*64kB (UEM) 309*128kB (UEM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2797460kB Aug 1 11:10:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:10:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:10:14 oak-gw06 kernel: 2117065 total pagecache pages Aug 1 11:10:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:10:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:10:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:10:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:10:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:10:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:10:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:15:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 11:15:14 oak-gw06 kernel: CPU: 2 PID: 12075 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:15:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:15:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:15:14 oak-gw06 kernel: 00000000000080d0 00000000cccc9fe3 ffff8800b2bb7858 ffffffff8168662f Aug 1 11:15:14 oak-gw06 kernel: ffff8800b2bb78e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:15:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800b2bb78b8 00000000cccc9fe3 Aug 1 11:15:14 oak-gw06 kernel: Call Trace: Aug 1 11:15:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:15:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:15:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:15:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:15:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 11:15:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 11:15:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:15:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:15:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:15:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:15:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:15:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:15:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:15:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:15:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:15:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:15:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:15:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:15:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:15:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:15:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:15:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:15:14 oak-gw06 kernel: Mem-Info: Aug 1 11:15:14 oak-gw06 kernel: active_anon:20265 inactive_anon:41519 isolated_anon:0#012 active_file:1901118 inactive_file:178158 isolated_file:0#012 unevictable:0 dirty:1852 writeback:889 unstable:0#012 slab_reclaimable:39428 slab_unreclaimable:827408#012 mapped:5861 shmem:39000 pagetables:1634 bounce:0#012 free:965233 free_pcp:51 free_cma:0 Aug 1 11:15:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:15:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:15:14 oak-gw06 kernel: Node 0 DMA32 free:1243284kB min:11976kB low:14968kB high:17964kB active_anon:13328kB inactive_anon:31248kB active_file:849328kB inactive_file:124168kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:2268kB shmem:31276kB slab_reclaimable:27760kB slab_unreclaimable:566044kB kernel_stack:1024kB pagetables:1048kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:15:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:15:14 oak-gw06 kernel: Node 0 Normal free:2600980kB min:55536kB low:69420kB high:83304kB active_anon:68252kB inactive_anon:134828kB active_file:6755664kB inactive_file:588464kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:7408kB writeback:3556kB mapped:21176kB shmem:124724kB slab_reclaimable:129952kB slab_unreclaimable:2743572kB kernel_stack:4736kB pagetables:5488kB unstable:0kB bounce:0kB free_pcp:800kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:15:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:15:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:15:14 oak-gw06 kernel: Node 0 DMA32: 18564*4kB (UEM) 19741*8kB (UEM) 17996*16kB (UEM) 14505*32kB (UEM) 3469*64kB (UEM) 285*128kB (UEM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1243288kB Aug 1 11:15:14 oak-gw06 kernel: Node 0 Normal: 16282*4kB (UEM) 75230*8kB (UEM) 56163*16kB (UEM) 22095*32kB (UEM) 4361*64kB (UEM) 376*128kB (UEM) 5*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2601128kB Aug 1 11:15:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:15:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:15:14 oak-gw06 kernel: 2118339 total pagecache pages Aug 1 11:15:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:15:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:15:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:15:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:15:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:15:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:15:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:15:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 11:15:14 oak-gw06 kernel: CPU: 2 PID: 12075 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:15:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:15:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:15:14 oak-gw06 kernel: 00000000000080d0 00000000cccc9fe3 ffff8800b2bb7808 ffffffff8168662f Aug 1 11:15:14 oak-gw06 kernel: ffff8800b2bb7898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:15:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800b2bb7868 00000000cccc9fe3 Aug 1 11:15:14 oak-gw06 kernel: Call Trace: Aug 1 11:15:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:15:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:15:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:15:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:15:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 11:15:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 11:15:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 11:15:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 11:15:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:15:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:15:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:15:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:15:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:15:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:15:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:15:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:15:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:15:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:15:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:15:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:15:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:15:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:15:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:15:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:15:14 oak-gw06 kernel: Mem-Info: Aug 1 11:15:14 oak-gw06 kernel: active_anon:21175 inactive_anon:41519 isolated_anon:0#012 active_file:1901183 inactive_file:178158 isolated_file:0#012 unevictable:0 dirty:1852 writeback:695 unstable:0#012 slab_reclaimable:39428 slab_unreclaimable:827408#012 mapped:5861 shmem:39000 pagetables:1634 bounce:0#012 free:964041 free_pcp:132 free_cma:0 Aug 1 11:15:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:15:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:15:14 oak-gw06 kernel: Node 0 DMA32 free:1243284kB min:11976kB low:14968kB high:17964kB active_anon:13328kB inactive_anon:31248kB active_file:849328kB inactive_file:124168kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:2268kB shmem:31276kB slab_reclaimable:27760kB slab_unreclaimable:566044kB kernel_stack:1024kB pagetables:1048kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:15:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:15:14 oak-gw06 kernel: Node 0 Normal free:2599080kB min:55536kB low:69420kB high:83304kB active_anon:67992kB inactive_anon:134828kB active_file:6756184kB inactive_file:588724kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:7020kB writeback:3556kB mapped:21176kB shmem:124724kB slab_reclaimable:129952kB slab_unreclaimable:2743572kB kernel_stack:4736kB pagetables:5488kB unstable:0kB bounce:0kB free_pcp:648kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:15:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:15:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:15:14 oak-gw06 kernel: Node 0 DMA32: 18564*4kB (UEM) 19741*8kB (UEM) 17996*16kB (UEM) 14505*32kB (UEM) 3469*64kB (UEM) 285*128kB (UEM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1243288kB Aug 1 11:15:14 oak-gw06 kernel: Node 0 Normal: 15991*4kB (UEM) 75183*8kB (UEM) 56094*16kB (UEM) 22094*32kB (UEM) 4361*64kB (UEM) 376*128kB (UEM) 5*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2598452kB Aug 1 11:15:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:15:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:15:14 oak-gw06 kernel: 2118436 total pagecache pages Aug 1 11:15:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:15:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:15:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:15:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:15:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:15:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:15:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:20:14 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 11:20:14 oak-gw06 kernel: CPU: 2 PID: 12466 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:20:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:20:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:20:14 oak-gw06 kernel: 00000000000080d0 000000005510278c ffff8801ecbcf858 ffffffff8168662f Aug 1 11:20:14 oak-gw06 kernel: ffff8801ecbcf8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:20:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801ecbcf8b8 000000005510278c Aug 1 11:20:14 oak-gw06 kernel: Call Trace: Aug 1 11:20:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:20:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:20:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:20:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:20:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 11:20:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 11:20:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:20:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:20:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:20:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:20:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:20:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:20:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:20:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:20:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:20:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:20:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:20:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:20:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:20:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:20:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:20:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:20:14 oak-gw06 kernel: Mem-Info: Aug 1 11:20:14 oak-gw06 kernel: active_anon:16012 inactive_anon:41519 isolated_anon:0#012 active_file:1857881 inactive_file:172857 isolated_file:0#012 unevictable:0 dirty:2628 writeback:62 unstable:0#012 slab_reclaimable:39506 slab_unreclaimable:833720#012 mapped:5689 shmem:39000 pagetables:1320 bounce:0#012 free:1012899 free_pcp:32 free_cma:0 Aug 1 11:20:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:20:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:20:14 oak-gw06 kernel: Node 0 DMA32 free:1421500kB min:11976kB low:14968kB high:17964kB active_anon:13160kB inactive_anon:31248kB active_file:673564kB inactive_file:121232kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:280kB writeback:0kB mapped:2344kB shmem:31276kB slab_reclaimable:27716kB slab_unreclaimable:563920kB kernel_stack:1040kB pagetables:1048kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:20:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:20:14 oak-gw06 kernel: Node 0 Normal free:2614204kB min:55536kB low:69420kB high:83304kB active_anon:50888kB inactive_anon:134828kB active_file:6757960kB inactive_file:570196kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10232kB writeback:248kB mapped:20412kB shmem:124724kB slab_reclaimable:130308kB slab_unreclaimable:2770944kB kernel_stack:4672kB pagetables:4232kB unstable:0kB bounce:0kB free_pcp:324kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:20:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:20:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:20:14 oak-gw06 kernel: Node 0 DMA32: 24079*4kB (UEM) 24156*8kB (UEM) 23036*16kB (UEM) 14872*32kB (UEM) 3797*64kB (UEM) 345*128kB (UEM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1421980kB Aug 1 11:20:14 oak-gw06 kernel: Node 0 Normal: 50364*4kB (UEM) 73555*8kB (UEM) 50978*16kB (UEM) 21204*32kB (UEM) 4361*64kB (UEM) 390*128kB (UEM) 5*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2614376kB Aug 1 11:20:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:20:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:20:14 oak-gw06 kernel: 2069671 total pagecache pages Aug 1 11:20:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:20:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:20:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:20:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:20:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:20:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:20:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:20:14 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 11:20:14 oak-gw06 kernel: CPU: 2 PID: 12466 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:20:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:20:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:20:14 oak-gw06 kernel: 00000000000080d0 000000005510278c ffff8801ecbcf808 ffffffff8168662f Aug 1 11:20:14 oak-gw06 kernel: ffff8801ecbcf898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:20:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801ecbcf868 000000005510278c Aug 1 11:20:14 oak-gw06 kernel: Call Trace: Aug 1 11:20:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:20:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:20:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:20:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:20:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 11:20:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 11:20:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 11:20:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 11:20:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:20:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:20:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:20:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:20:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:20:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:20:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:20:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:20:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:20:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:20:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:20:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:20:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:20:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:20:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:20:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:20:14 oak-gw06 kernel: Mem-Info: Aug 1 11:20:14 oak-gw06 kernel: active_anon:16012 inactive_anon:41519 isolated_anon:0#012 active_file:1857816 inactive_file:172857 isolated_file:0#012 unevictable:0 dirty:2628 writeback:62 unstable:0#012 slab_reclaimable:39506 slab_unreclaimable:833720#012 mapped:5689 shmem:39000 pagetables:1320 bounce:0#012 free:1012972 free_pcp:31 free_cma:0 Aug 1 11:20:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:20:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:20:14 oak-gw06 kernel: Node 0 DMA32 free:1421500kB min:11976kB low:14968kB high:17964kB active_anon:13160kB inactive_anon:31248kB active_file:673564kB inactive_file:121232kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:280kB writeback:0kB mapped:2344kB shmem:31276kB slab_reclaimable:27716kB slab_unreclaimable:563920kB kernel_stack:1040kB pagetables:1048kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:20:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:20:14 oak-gw06 kernel: Node 0 Normal free:2614496kB min:55536kB low:69420kB high:83304kB active_anon:50888kB inactive_anon:134828kB active_file:6757700kB inactive_file:570196kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10232kB writeback:248kB mapped:20412kB shmem:124724kB slab_reclaimable:130308kB slab_unreclaimable:2770944kB kernel_stack:4672kB pagetables:4232kB unstable:0kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:20:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:20:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:20:14 oak-gw06 kernel: Node 0 DMA32: 24082*4kB (UEM) 24178*8kB (UEM) 23031*16kB (UEM) 14876*32kB (UEM) 3798*64kB (UEM) 345*128kB (UEM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1422280kB Aug 1 11:20:14 oak-gw06 kernel: Node 0 Normal: 50366*4kB (UEM) 73581*8kB (UEM) 50978*16kB (UEM) 21204*32kB (UEM) 4361*64kB (UEM) 390*128kB (UEM) 5*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2614592kB Aug 1 11:20:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:20:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:20:14 oak-gw06 kernel: 2069574 total pagecache pages Aug 1 11:20:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:20:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:20:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:20:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:20:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:20:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:20:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:25:14 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 1 11:25:14 oak-gw06 kernel: CPU: 2 PID: 12564 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:25:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:25:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:25:14 oak-gw06 kernel: 00000000000080d0 00000000739e27c3 ffff88029c87b858 ffffffff8168662f Aug 1 11:25:14 oak-gw06 kernel: ffff88029c87b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:25:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88029c87b8b8 00000000739e27c3 Aug 1 11:25:14 oak-gw06 kernel: Call Trace: Aug 1 11:25:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:25:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:25:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:25:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:25:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 11:25:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 11:25:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:25:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:25:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:25:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:25:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:25:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:25:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:25:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:25:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:25:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:25:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:25:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:25:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:25:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:25:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:25:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:25:14 oak-gw06 kernel: Mem-Info: Aug 1 11:25:14 oak-gw06 kernel: active_anon:21945 inactive_anon:41519 isolated_anon:0#012 active_file:1839077 inactive_file:150074 isolated_file:0#012 unevictable:0 dirty:481 writeback:24 unstable:0#012 slab_reclaimable:40122 slab_unreclaimable:841949#012 mapped:5881 shmem:39000 pagetables:1609 bounce:0#012 free:1039004 free_pcp:205 free_cma:0 Aug 1 11:25:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:25:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:25:14 oak-gw06 kernel: Node 0 DMA32 free:1573948kB min:11976kB low:14968kB high:17964kB active_anon:14724kB inactive_anon:31248kB active_file:525576kB inactive_file:106536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:16kB writeback:8kB mapped:2384kB shmem:31276kB slab_reclaimable:27740kB slab_unreclaimable:562300kB kernel_stack:1024kB pagetables:1180kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:25:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:25:14 oak-gw06 kernel: Node 0 Normal free:2565896kB min:55536kB low:69420kB high:83304kB active_anon:73316kB inactive_anon:134828kB active_file:6830732kB inactive_file:493760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:1908kB writeback:88kB mapped:21140kB shmem:124724kB slab_reclaimable:132748kB slab_unreclaimable:2805480kB kernel_stack:4736kB pagetables:5256kB unstable:0kB bounce:0kB free_pcp:748kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:25:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:25:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:25:14 oak-gw06 kernel: Node 0 DMA32: 34002*4kB (UEM) 29537*8kB (UEM) 25032*16kB (UEM) 15235*32kB (UEM) 4083*64kB (UEM) 408*128kB (UEM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1574896kB Aug 1 11:25:14 oak-gw06 kernel: Node 0 Normal: 71941*4kB (UEM) 67198*8kB (UEM) 47333*16kB (UM) 20392*32kB (UEM) 4354*64kB (UEM) 394*128kB (UEM) 5*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2565588kB Aug 1 11:25:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:25:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:25:14 oak-gw06 kernel: 2028152 total pagecache pages Aug 1 11:25:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:25:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:25:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:25:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:25:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:25:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:25:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:25:14 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 1 11:25:14 oak-gw06 kernel: CPU: 2 PID: 12564 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:25:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:25:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:25:14 oak-gw06 kernel: 00000000000080d0 00000000739e27c3 ffff88029c87b808 ffffffff8168662f Aug 1 11:25:14 oak-gw06 kernel: ffff88029c87b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:25:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88029c87b868 00000000739e27c3 Aug 1 11:25:14 oak-gw06 kernel: Call Trace: Aug 1 11:25:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:25:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:25:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:25:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:25:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 11:25:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 11:25:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 11:25:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 11:25:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:25:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:25:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:25:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:25:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:25:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:25:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:25:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:25:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:25:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:25:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:25:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:25:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:25:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:25:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:25:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:25:14 oak-gw06 kernel: Mem-Info: Aug 1 11:25:14 oak-gw06 kernel: active_anon:22010 inactive_anon:41519 isolated_anon:0#012 active_file:1839077 inactive_file:150009 isolated_file:0#012 unevictable:0 dirty:481 writeback:24 unstable:0#012 slab_reclaimable:40122 slab_unreclaimable:841949#012 mapped:5881 shmem:39000 pagetables:1609 bounce:0#012 free:1039166 free_pcp:90 free_cma:0 Aug 1 11:25:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:25:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:25:14 oak-gw06 kernel: Node 0 DMA32 free:1573948kB min:11976kB low:14968kB high:17964kB active_anon:14724kB inactive_anon:31248kB active_file:525576kB inactive_file:106536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:16kB writeback:8kB mapped:2384kB shmem:31276kB slab_reclaimable:27740kB slab_unreclaimable:562300kB kernel_stack:1024kB pagetables:1180kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:25:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:25:14 oak-gw06 kernel: Node 0 Normal free:2566824kB min:55536kB low:69420kB high:83304kB active_anon:73316kB inactive_anon:134828kB active_file:6830732kB inactive_file:493500kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:1908kB writeback:88kB mapped:21140kB shmem:124724kB slab_reclaimable:132748kB slab_unreclaimable:2805480kB kernel_stack:4736kB pagetables:5256kB unstable:0kB bounce:0kB free_pcp:344kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:25:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:25:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:25:14 oak-gw06 kernel: Node 0 DMA32: 34004*4kB (UEM) 29537*8kB (UEM) 25032*16kB (UEM) 15235*32kB (UEM) 4083*64kB (UEM) 408*128kB (UEM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1574904kB Aug 1 11:25:14 oak-gw06 kernel: Node 0 Normal: 72072*4kB (UEM) 67201*8kB (UEM) 47329*16kB (UM) 20392*32kB (UEM) 4354*64kB (UEM) 394*128kB (UEM) 5*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2566072kB Aug 1 11:25:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:25:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:25:14 oak-gw06 kernel: 2028055 total pagecache pages Aug 1 11:25:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:25:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:25:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:25:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:25:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:25:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:25:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:30:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 11:30:14 oak-gw06 kernel: CPU: 2 PID: 12728 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:30:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:30:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:30:14 oak-gw06 kernel: 00000000000080d0 00000000198dde2b ffff8800bac23858 ffffffff8168662f Aug 1 11:30:14 oak-gw06 kernel: ffff8800bac238e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 1 11:30:14 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff8800bac238e8 00000000198dde2b Aug 1 11:30:14 oak-gw06 kernel: Call Trace: Aug 1 11:30:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:30:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:30:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:30:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:30:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 11:30:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 11:30:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:30:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:30:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:30:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:30:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:30:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:30:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:30:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:30:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:30:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:30:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:30:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:30:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:30:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:30:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:30:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:30:14 oak-gw06 kernel: Mem-Info: Aug 1 11:30:14 oak-gw06 kernel: active_anon:26735 inactive_anon:41519 isolated_anon:0#012 active_file:2028199 inactive_file:4669 isolated_file:0#012 unevictable:0 dirty:33856 writeback:8990 unstable:0#012 slab_reclaimable:37675 slab_unreclaimable:842870#012 mapped:5892 shmem:39000 pagetables:1630 bounce:0#012 free:889650 free_pcp:544 free_cma:0 Aug 1 11:30:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:30:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:30:14 oak-gw06 kernel: Node 0 DMA32 free:778396kB min:11976kB low:14968kB high:17964kB active_anon:15084kB inactive_anon:31248kB active_file:1341280kB inactive_file:1444kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:22480kB writeback:4692kB mapped:2428kB shmem:31276kB slab_reclaimable:26880kB slab_unreclaimable:572076kB kernel_stack:1040kB pagetables:1108kB unstable:0kB bounce:0kB free_pcp:888kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:30:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:30:14 oak-gw06 kernel: Node 0 Normal free:2762944kB min:55536kB low:69420kB high:83304kB active_anon:91336kB inactive_anon:134828kB active_file:6779576kB inactive_file:11516kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:115372kB writeback:31268kB mapped:21140kB shmem:124724kB slab_reclaimable:123820kB slab_unreclaimable:2799388kB kernel_stack:4672kB pagetables:5412kB unstable:0kB bounce:0kB free_pcp:1240kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:30:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:30:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:30:14 oak-gw06 kernel: Node 0 DMA32: 2411*4kB (UEM) 2508*8kB (UEM) 3949*16kB (UEM) 11065*32kB (UEM) 4234*64kB (UEM) 461*128kB (UEM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 777980kB Aug 1 11:30:14 oak-gw06 kernel: Node 0 Normal: 28830*4kB (UEM) 81889*8kB (UEM) 59651*16kB (UEM) 18927*32kB (UEM) 5331*64kB (UEM) 673*128kB (UEM) 20*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2762960kB Aug 1 11:30:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:30:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:30:14 oak-gw06 kernel: 2072288 total pagecache pages Aug 1 11:30:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:30:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:30:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:30:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:30:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:30:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:30:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:30:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 11:30:14 oak-gw06 kernel: CPU: 2 PID: 12728 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:30:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:30:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:30:14 oak-gw06 kernel: 00000000000080d0 00000000198dde2b ffff8800bac23808 ffffffff8168662f Aug 1 11:30:14 oak-gw06 kernel: ffff8800bac23898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:30:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800bac23868 00000000198dde2b Aug 1 11:30:14 oak-gw06 kernel: Call Trace: Aug 1 11:30:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:30:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:30:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:30:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:30:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 11:30:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 11:30:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 11:30:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 11:30:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:30:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:30:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:30:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:30:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:30:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:30:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:30:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:30:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:30:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:30:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:30:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:30:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:30:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:30:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:30:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:30:14 oak-gw06 kernel: Mem-Info: Aug 1 11:30:14 oak-gw06 kernel: active_anon:26605 inactive_anon:41519 isolated_anon:0#012 active_file:2031551 inactive_file:1809 isolated_file:0#012 unevictable:0 dirty:34463 writeback:8990 unstable:0#012 slab_reclaimable:37675 slab_unreclaimable:842870#012 mapped:5892 shmem:39000 pagetables:1630 bounce:0#012 free:889679 free_pcp:31 free_cma:0 Aug 1 11:30:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:30:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:30:14 oak-gw06 kernel: Node 0 DMA32 free:779100kB min:11976kB low:14968kB high:17964kB active_anon:15084kB inactive_anon:31248kB active_file:1341688kB inactive_file:660kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:22480kB writeback:4692kB mapped:2428kB shmem:31276kB slab_reclaimable:26880kB slab_unreclaimable:572076kB kernel_stack:1040kB pagetables:1108kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:30:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:30:15 oak-gw06 kernel: Node 0 Normal free:2762896kB min:55536kB low:69420kB high:83304kB active_anon:91336kB inactive_anon:134828kB active_file:6784516kB inactive_file:6576kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:115372kB writeback:31268kB mapped:21140kB shmem:124724kB slab_reclaimable:123820kB slab_unreclaimable:2799388kB kernel_stack:4672kB pagetables:5412kB unstable:0kB bounce:0kB free_pcp:740kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:30:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:30:15 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:30:15 oak-gw06 kernel: Node 0 DMA32: 2796*4kB (UEM) 2519*8kB (UEM) 3981*16kB (UEM) 11065*32kB (UEM) 4234*64kB (UEM) 461*128kB (UEM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 780120kB Aug 1 11:30:15 oak-gw06 kernel: Node 0 Normal: 28881*4kB (UEM) 81906*8kB (UEM) 59655*16kB (UEM) 18930*32kB (UEM) 5331*64kB (UEM) 673*128kB (UEM) 20*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2763460kB Aug 1 11:30:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:30:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:30:15 oak-gw06 kernel: 2072148 total pagecache pages Aug 1 11:30:15 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:30:15 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:30:15 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:30:15 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:30:15 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:30:15 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:30:15 oak-gw06 kernel: 127313 pages reserved Aug 1 11:35:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 11:35:14 oak-gw06 kernel: CPU: 2 PID: 12728 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:35:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:35:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:35:14 oak-gw06 kernel: 00000000000080d0 00000000198dde2b ffff8800bac23858 ffffffff8168662f Aug 1 11:35:14 oak-gw06 kernel: ffff8800bac238e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:35:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800bac238b8 00000000198dde2b Aug 1 11:35:14 oak-gw06 kernel: Call Trace: Aug 1 11:35:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:35:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:35:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:35:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:35:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 11:35:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 11:35:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:35:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:35:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:35:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:35:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:35:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:35:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:35:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:35:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:35:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:35:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:35:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:35:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:35:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:35:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:35:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:35:14 oak-gw06 kernel: Mem-Info: Aug 1 11:35:14 oak-gw06 kernel: active_anon:29276 inactive_anon:41519 isolated_anon:0#012 active_file:2013038 inactive_file:23795 isolated_file:0#012 unevictable:0 dirty:31689 writeback:13258 unstable:0#012 slab_reclaimable:37177 slab_unreclaimable:844859#012 mapped:5904 shmem:39000 pagetables:1642 bounce:0#012 free:933040 free_pcp:268 free_cma:0 Aug 1 11:35:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:35:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:35:14 oak-gw06 kernel: Node 0 DMA32 free:711120kB min:11976kB low:14968kB high:17964kB active_anon:19436kB inactive_anon:31248kB active_file:1415572kB inactive_file:15536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:23252kB writeback:9720kB mapped:2476kB shmem:31276kB slab_reclaimable:26736kB slab_unreclaimable:574160kB kernel_stack:1056kB pagetables:1712kB unstable:0kB bounce:0kB free_pcp:464kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:35:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:35:14 oak-gw06 kernel: Node 0 Normal free:2998984kB min:55536kB low:69420kB high:83304kB active_anon:97668kB inactive_anon:134828kB active_file:6636580kB inactive_file:82244kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:106220kB writeback:43312kB mapped:21140kB shmem:124724kB slab_reclaimable:121972kB slab_unreclaimable:2805260kB kernel_stack:4640kB pagetables:4856kB unstable:0kB bounce:0kB free_pcp:1696kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:35:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:35:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:35:14 oak-gw06 kernel: Node 0 DMA32: 4734*4kB (UEM) 4129*8kB (UEM) 1246*16kB (UEM) 9328*32kB (UEM) 4296*64kB (UEM) 489*128kB (UEM) 7*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 709728kB Aug 1 11:35:14 oak-gw06 kernel: Node 0 Normal: 28507*4kB (UEM) 75208*8kB (UEM) 62318*16kB (UEM) 23267*32kB (UEM) 6150*64kB (UEM) 1018*128kB (UEM) 52*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 2995052kB Aug 1 11:35:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:35:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:35:14 oak-gw06 kernel: 2073299 total pagecache pages Aug 1 11:35:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:35:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:35:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:35:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:35:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:35:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:35:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:35:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 11:35:14 oak-gw06 kernel: CPU: 1 PID: 12728 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:35:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:35:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:35:14 oak-gw06 kernel: 00000000000080d0 00000000198dde2b ffff8800bac23808 ffffffff8168662f Aug 1 11:35:14 oak-gw06 kernel: ffff8800bac23898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:35:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800bac23868 00000000198dde2b Aug 1 11:35:14 oak-gw06 kernel: Call Trace: Aug 1 11:35:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:35:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:35:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:35:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:35:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 11:35:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 11:35:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 11:35:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 11:35:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:35:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:35:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:35:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:35:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:35:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:35:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:35:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:35:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:35:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:35:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:35:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:35:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:35:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:35:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:35:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:35:14 oak-gw06 kernel: Mem-Info: Aug 1 11:35:14 oak-gw06 kernel: active_anon:29276 inactive_anon:43534 isolated_anon:0#012 active_file:2013038 inactive_file:24893 isolated_file:0#012 unevictable:0 dirty:32799 writeback:13258 unstable:0#012 slab_reclaimable:37177 slab_unreclaimable:844859#012 mapped:5904 shmem:41037 pagetables:1642 bounce:0#012 free:926555 free_pcp:154 free_cma:0 Aug 1 11:35:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:35:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:35:14 oak-gw06 kernel: Node 0 DMA32 free:706828kB min:11976kB low:14968kB high:17964kB active_anon:19436kB inactive_anon:31248kB active_file:1415572kB inactive_file:16288kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:24372kB writeback:9720kB mapped:2476kB shmem:31276kB slab_reclaimable:26736kB slab_unreclaimable:574160kB kernel_stack:1056kB pagetables:1712kB unstable:0kB bounce:0kB free_pcp:64kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:35:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:35:14 oak-gw06 kernel: Node 0 Normal free:2976396kB min:55536kB low:69420kB high:83304kB active_anon:97668kB inactive_anon:142888kB active_file:6636580kB inactive_file:84064kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:108548kB writeback:43312kB mapped:21140kB shmem:132872kB slab_reclaimable:121972kB slab_unreclaimable:2805260kB kernel_stack:4640kB pagetables:4856kB unstable:0kB bounce:0kB free_pcp:1168kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:35:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:35:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:35:14 oak-gw06 kernel: Node 0 DMA32: 4658*4kB (UEM) 4128*8kB (UEM) 1174*16kB (UEM) 9328*32kB (UEM) 4296*64kB (UEM) 489*128kB (UEM) 7*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 708264kB Aug 1 11:35:14 oak-gw06 kernel: Node 0 Normal: 26224*4kB (EM) 74393*8kB (UEM) 61867*16kB (UEM) 23267*32kB (UEM) 6150*64kB (UEM) 1018*128kB (UEM) 52*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 2972184kB Aug 1 11:35:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:35:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:35:14 oak-gw06 kernel: 2075961 total pagecache pages Aug 1 11:35:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:35:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:35:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:35:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:35:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:35:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:35:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:40:14 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 1 11:40:14 oak-gw06 kernel: CPU: 2 PID: 12564 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:40:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:40:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:40:14 oak-gw06 kernel: 00000000000080d0 00000000739e27c3 ffff88029c87b858 ffffffff8168662f Aug 1 11:40:14 oak-gw06 kernel: ffff88029c87b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:40:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88029c87b8b8 00000000739e27c3 Aug 1 11:40:14 oak-gw06 kernel: Call Trace: Aug 1 11:40:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:40:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:40:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:40:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:40:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 11:40:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 11:40:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:40:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:40:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:40:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:40:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:40:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:40:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:40:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:40:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:40:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:40:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:40:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:40:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:40:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:40:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:40:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:40:14 oak-gw06 kernel: Mem-Info: Aug 1 11:40:14 oak-gw06 kernel: active_anon:21870 inactive_anon:43567 isolated_anon:0#012 active_file:2026237 inactive_file:34698 isolated_file:0#012 unevictable:0 dirty:3796 writeback:194 unstable:0#012 slab_reclaimable:37560 slab_unreclaimable:844199#012 mapped:5918 shmem:41048 pagetables:1604 bounce:0#012 free:965284 free_pcp:189 free_cma:0 Aug 1 11:40:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:40:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:40:14 oak-gw06 kernel: Node 0 DMA32 free:1040000kB min:11976kB low:14968kB high:17964kB active_anon:13364kB inactive_anon:31248kB active_file:1143612kB inactive_file:17652kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:580kB writeback:0kB mapped:2508kB shmem:31276kB slab_reclaimable:26668kB slab_unreclaimable:565916kB kernel_stack:1024kB pagetables:1080kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:40:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:40:14 oak-gw06 kernel: Node 0 Normal free:2804336kB min:55536kB low:69420kB high:83304kB active_anon:74116kB inactive_anon:143020kB active_file:6961336kB inactive_file:121140kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:14604kB writeback:776kB mapped:21164kB shmem:132916kB slab_reclaimable:123572kB slab_unreclaimable:2810864kB kernel_stack:4736kB pagetables:5336kB unstable:0kB bounce:0kB free_pcp:1200kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:40:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:40:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:40:14 oak-gw06 kernel: Node 0 DMA32: 17557*4kB (UEM) 17090*8kB (UEM) 10116*16kB (UEM) 10286*32kB (UEM) 4322*64kB (UEM) 513*128kB (UEM) 7*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1042020kB Aug 1 11:40:14 oak-gw06 kernel: Node 0 Normal: 23342*4kB (UEM) 60838*8kB (UEM) 56622*16kB (UEM) 22742*32kB (UEM) 6441*64kB (UEM) 1228*128kB (UEM) 82*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 2804680kB Aug 1 11:40:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:40:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:40:14 oak-gw06 kernel: 2102080 total pagecache pages Aug 1 11:40:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:40:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:40:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:40:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:40:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:40:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:40:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:40:14 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 1 11:40:14 oak-gw06 kernel: CPU: 5 PID: 12564 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:40:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:40:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:40:14 oak-gw06 kernel: 00000000000080d0 00000000739e27c3 ffff88029c87b808 ffffffff8168662f Aug 1 11:40:14 oak-gw06 kernel: ffff88029c87b898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 1 11:40:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88029c87b868 00000000739e27c3 Aug 1 11:40:14 oak-gw06 kernel: Call Trace: Aug 1 11:40:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:40:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:40:14 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 1 11:40:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:40:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:40:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 11:40:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 11:40:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 11:40:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 11:40:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:40:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:40:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:40:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:40:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:40:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:40:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:40:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:40:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:40:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:40:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:40:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:40:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:40:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:40:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:40:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:40:14 oak-gw06 kernel: Mem-Info: Aug 1 11:40:14 oak-gw06 kernel: active_anon:21870 inactive_anon:43567 isolated_anon:0#012 active_file:2026237 inactive_file:34698 isolated_file:0#012 unevictable:0 dirty:3796 writeback:194 unstable:0#012 slab_reclaimable:37560 slab_unreclaimable:844199#012 mapped:5918 shmem:41048 pagetables:1604 bounce:0#012 free:965120 free_pcp:248 free_cma:0 Aug 1 11:40:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:40:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:40:14 oak-gw06 kernel: Node 0 DMA32 free:1040000kB min:11976kB low:14968kB high:17964kB active_anon:13364kB inactive_anon:31248kB active_file:1143612kB inactive_file:17652kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:580kB writeback:0kB mapped:2508kB shmem:31276kB slab_reclaimable:26668kB slab_unreclaimable:565916kB kernel_stack:1024kB pagetables:1080kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:40:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:40:14 oak-gw06 kernel: Node 0 Normal free:2804316kB min:55536kB low:69420kB high:83304kB active_anon:74376kB inactive_anon:143020kB active_file:6961336kB inactive_file:121140kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:14604kB writeback:776kB mapped:21164kB shmem:132916kB slab_reclaimable:123572kB slab_unreclaimable:2810864kB kernel_stack:4736kB pagetables:5336kB unstable:0kB bounce:0kB free_pcp:1000kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:40:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:40:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:40:14 oak-gw06 kernel: Node 0 DMA32: 17558*4kB (UEM) 17090*8kB (UEM) 10116*16kB (UEM) 10286*32kB (UEM) 4322*64kB (UEM) 513*128kB (UEM) 7*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1042024kB Aug 1 11:40:14 oak-gw06 kernel: Node 0 Normal: 23422*4kB (UEM) 60838*8kB (UEM) 56619*16kB (UEM) 22740*32kB (UEM) 6441*64kB (UEM) 1228*128kB (UEM) 82*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 2804888kB Aug 1 11:40:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:40:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:40:14 oak-gw06 kernel: 2102080 total pagecache pages Aug 1 11:40:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:40:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:40:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:40:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:40:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:40:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:40:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:45:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 11:45:14 oak-gw06 kernel: CPU: 2 PID: 12728 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:45:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:45:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:45:14 oak-gw06 kernel: 00000000000080d0 00000000198dde2b ffff8800bac23858 ffffffff8168662f Aug 1 11:45:14 oak-gw06 kernel: ffff8800bac238e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:45:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800bac238b8 00000000198dde2b Aug 1 11:45:14 oak-gw06 kernel: Call Trace: Aug 1 11:45:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:45:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:45:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:45:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:45:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 11:45:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 11:45:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:45:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:45:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:45:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:45:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:45:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:45:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:45:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:45:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:45:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:45:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:45:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:45:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:45:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:45:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:45:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:45:14 oak-gw06 kernel: Mem-Info: Aug 1 11:45:14 oak-gw06 kernel: active_anon:28368 inactive_anon:43567 isolated_anon:0#012 active_file:1306354 inactive_file:919903 isolated_file:0#012 unevictable:0 dirty:795 writeback:1696 unstable:0#012 slab_reclaimable:38025 slab_unreclaimable:846043#012 mapped:5940 shmem:41048 pagetables:1648 bounce:0#012 free:786648 free_pcp:1590 free_cma:0 Aug 1 11:45:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:45:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:45:14 oak-gw06 kernel: Node 0 DMA32 free:742124kB min:11976kB low:14968kB high:17964kB active_anon:14908kB inactive_anon:31248kB active_file:765144kB inactive_file:676760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2108kB writeback:464kB mapped:2508kB shmem:31276kB slab_reclaimable:26920kB slab_unreclaimable:571288kB kernel_stack:1024kB pagetables:1052kB unstable:0kB bounce:0kB free_pcp:3660kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:45:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:45:14 oak-gw06 kernel: Node 0 Normal free:2407564kB min:55536kB low:69420kB high:83304kB active_anon:98564kB inactive_anon:143020kB active_file:4460272kB inactive_file:2984996kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:2408kB writeback:2700kB mapped:21252kB shmem:132916kB slab_reclaimable:125180kB slab_unreclaimable:2812596kB kernel_stack:4672kB pagetables:5540kB unstable:0kB bounce:0kB free_pcp:3216kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:45:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:45:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:45:14 oak-gw06 kernel: Node 0 DMA32: 5566*4kB (UEM) 4235*8kB (UEM) 2061*16kB (UEM) 9402*32kB (UEM) 4414*64kB (UEM) 558*128kB (UEM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 745952kB Aug 1 11:45:14 oak-gw06 kernel: Node 0 Normal: 22188*4kB (UEM) 13446*8kB (UEM) 51497*16kB (UEM) 23152*32kB (UEM) 6775*64kB (UEM) 1394*128kB (UEM) 119*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 2404144kB Aug 1 11:45:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:45:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:45:14 oak-gw06 kernel: 2102445 total pagecache pages Aug 1 11:45:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:45:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:45:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:45:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:45:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:45:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:45:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:45:14 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 1 11:45:14 oak-gw06 kernel: CPU: 2 PID: 12728 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:45:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:45:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:45:14 oak-gw06 kernel: 00000000000080d0 00000000198dde2b ffff8800bac23808 ffffffff8168662f Aug 1 11:45:14 oak-gw06 kernel: ffff8800bac23898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:45:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800bac23868 00000000198dde2b Aug 1 11:45:14 oak-gw06 kernel: Call Trace: Aug 1 11:45:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:45:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:45:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:45:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:45:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 11:45:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 11:45:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 11:45:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 11:45:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:45:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:45:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:45:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:45:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:45:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:45:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:45:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:45:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:45:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:45:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:45:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:45:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:45:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:45:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:45:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:45:14 oak-gw06 kernel: Mem-Info: Aug 1 11:45:14 oak-gw06 kernel: active_anon:28368 inactive_anon:43567 isolated_anon:0#012 active_file:1306289 inactive_file:924129 isolated_file:0#012 unevictable:0 dirty:1043 writeback:1351 unstable:0#012 slab_reclaimable:38025 slab_unreclaimable:845975#012 mapped:5940 shmem:41048 pagetables:1648 bounce:0#012 free:786051 free_pcp:704 free_cma:0 Aug 1 11:45:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:45:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:45:14 oak-gw06 kernel: Node 0 DMA32 free:740188kB min:11976kB low:14968kB high:17964kB active_anon:14908kB inactive_anon:31248kB active_file:765144kB inactive_file:683528kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:988kB writeback:2144kB mapped:2508kB shmem:31276kB slab_reclaimable:26920kB slab_unreclaimable:571288kB kernel_stack:1024kB pagetables:1052kB unstable:0kB bounce:0kB free_pcp:1172kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:45:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:45:14 oak-gw06 kernel: Node 0 Normal free:2371132kB min:55536kB low:69420kB high:83304kB active_anon:98564kB inactive_anon:143020kB active_file:4460012kB inactive_file:3031796kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:2408kB writeback:5028kB mapped:21252kB shmem:132916kB slab_reclaimable:125180kB slab_unreclaimable:2812596kB kernel_stack:4672kB pagetables:5540kB unstable:0kB bounce:0kB free_pcp:2892kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:45:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:45:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:45:14 oak-gw06 kernel: Node 0 DMA32: 4313*4kB (UEM) 3743*8kB (UEM) 2128*16kB (UEM) 9400*32kB (UEM) 4414*64kB (UEM) 558*128kB (UEM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 738012kB Aug 1 11:45:14 oak-gw06 kernel: Node 0 Normal: 12464*4kB (UE) 12705*8kB (UEM) 51758*16kB (UEM) 23177*32kB (UEM) 6775*64kB (UEM) 1394*128kB (UEM) 119*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 2364296kB Aug 1 11:45:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:45:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:45:14 oak-gw06 kernel: 2100879 total pagecache pages Aug 1 11:45:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:45:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:45:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:45:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:45:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:45:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:45:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:50:14 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 11:50:14 oak-gw06 kernel: CPU: 6 PID: 12808 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:50:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:50:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:50:14 oak-gw06 kernel: 00000000000080d0 00000000c1caf35e ffff880408e0f858 ffffffff8168662f Aug 1 11:50:14 oak-gw06 kernel: ffff880408e0f8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:50:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880408e0f8b8 00000000c1caf35e Aug 1 11:50:14 oak-gw06 kernel: Call Trace: Aug 1 11:50:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:50:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:50:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:50:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:50:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 1 11:50:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 1 11:50:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:50:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:50:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:50:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:50:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:50:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:50:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:50:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:50:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:50:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:50:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:50:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:50:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:50:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:50:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:50:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:50:14 oak-gw06 kernel: Mem-Info: Aug 1 11:50:14 oak-gw06 kernel: active_anon:23023 inactive_anon:43567 isolated_anon:0#012 active_file:870359 inactive_file:1250080 isolated_file:0#012 unevictable:0 dirty:6121 writeback:3448 unstable:0#012 slab_reclaimable:37905 slab_unreclaimable:844470#012 mapped:5942 shmem:41048 pagetables:1626 bounce:0#012 free:868965 free_pcp:1229 free_cma:0 Aug 1 11:50:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:50:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:50:14 oak-gw06 kernel: Node 0 DMA32 free:743428kB min:11976kB low:14968kB high:17964kB active_anon:13936kB inactive_anon:31248kB active_file:527184kB inactive_file:889224kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:5776kB writeback:1144kB mapped:2516kB shmem:31276kB slab_reclaimable:26888kB slab_unreclaimable:570644kB kernel_stack:1024kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:2252kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:50:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:50:14 oak-gw06 kernel: Node 0 Normal free:2735272kB min:55536kB low:69420kB high:83304kB active_anon:78156kB inactive_anon:143020kB active_file:2954252kB inactive_file:4092084kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:21768kB writeback:13424kB mapped:21252kB shmem:132916kB slab_reclaimable:124732kB slab_unreclaimable:2807220kB kernel_stack:4656kB pagetables:5444kB unstable:0kB bounce:0kB free_pcp:2232kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:50:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:50:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:50:14 oak-gw06 kernel: Node 0 DMA32: 3810*4kB (UEM) 5867*8kB (UEM) 4726*16kB (UEM) 8192*32kB (UEM) 4339*64kB (UEM) 548*128kB (UEM) 9*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 750080kB Aug 1 11:50:14 oak-gw06 kernel: Node 0 Normal: 16059*4kB (UEM) 56313*8kB (UEM) 47947*16kB (UEM) 24846*32kB (UEM) 6940*64kB (UEM) 1449*128kB (UEM) 131*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 2740644kB Aug 1 11:50:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:50:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:50:14 oak-gw06 kernel: 2119977 total pagecache pages Aug 1 11:50:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:50:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:50:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:50:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:50:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:50:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:50:14 oak-gw06 kernel: 127313 pages reserved Aug 1 11:50:14 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 1 11:50:14 oak-gw06 kernel: CPU: 6 PID: 12808 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 1 11:50:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 1 11:50:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 1 11:50:14 oak-gw06 kernel: 00000000000080d0 00000000c1caf35e ffff880408e0f808 ffffffff8168662f Aug 1 11:50:14 oak-gw06 kernel: ffff880408e0f898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 1 11:50:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880408e0f868 00000000c1caf35e Aug 1 11:50:14 oak-gw06 kernel: Call Trace: Aug 1 11:50:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 1 11:50:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 1 11:50:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 1 11:50:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 1 11:50:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 1 11:50:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 1 11:50:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 1 11:50:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 1 11:50:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 1 11:50:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 1 11:50:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 1 11:50:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 1 11:50:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 1 11:50:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 1 11:50:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 1 11:50:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 1 11:50:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 1 11:50:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 1 11:50:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 1 11:50:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 1 11:50:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 1 11:50:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:50:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 1 11:50:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 1 11:50:14 oak-gw06 kernel: Mem-Info: Aug 1 11:50:14 oak-gw06 kernel: active_anon:23023 inactive_anon:43567 isolated_anon:0#012 active_file:870359 inactive_file:1247219 isolated_file:0#012 unevictable:0 dirty:6789 writeback:2629 unstable:0#012 slab_reclaimable:37905 slab_unreclaimable:844470#012 mapped:5942 shmem:41048 pagetables:1626 bounce:0#012 free:870287 free_pcp:241 free_cma:0 Aug 1 11:50:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 1 11:50:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 1 11:50:14 oak-gw06 kernel: Node 0 DMA32 free:744512kB min:11976kB low:14968kB high:17964kB active_anon:13936kB inactive_anon:31248kB active_file:527184kB inactive_file:889600kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:5216kB writeback:1144kB mapped:2516kB shmem:31276kB slab_reclaimable:26888kB slab_unreclaimable:570644kB kernel_stack:1024kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:384kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:50:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 1 11:50:14 oak-gw06 kernel: Node 0 Normal free:2704248kB min:55536kB low:69420kB high:83304kB active_anon:78416kB inactive_anon:143020kB active_file:2954252kB inactive_file:4114964kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:21380kB writeback:9932kB mapped:21252kB shmem:132916kB slab_reclaimable:124732kB slab_unreclaimable:2807220kB kernel_stack:4656kB pagetables:5444kB unstable:0kB bounce:0kB free_pcp:508kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 1 11:50:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 1 11:50:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 1 11:50:14 oak-gw06 kernel: Node 0 DMA32: 1604*4kB (UEM) 5776*8kB (UEM) 4677*16kB (UEM) 8246*32kB (UEM) 4339*64kB (UEM) 548*128kB (UEM) 9*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 741472kB Aug 1 11:50:14 oak-gw06 kernel: Node 0 Normal: 6187*4kB (UEM) 56075*8kB (UEM) 47687*16kB (UEM) 24847*32kB (UEM) 6940*64kB (UEM) 1449*128kB (UEM) 131*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 2695124kB Aug 1 11:50:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 1 11:50:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 11:50:14 oak-gw06 kernel: 2126831 total pagecache pages Aug 1 11:50:14 oak-gw06 kernel: 0 pages in swap cache Aug 1 11:50:14 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 1 11:50:14 oak-gw06 kernel: Free swap = 4194300kB Aug 1 11:50:14 oak-gw06 kernel: Total swap = 4194300kB Aug 1 11:50:14 oak-gw06 kernel: 4194203 pages RAM Aug 1 11:50:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 1 11:50:14 oak-gw06 kernel: 127313 pages reserved Aug 1 13:30:46 oak-gw06 kernel: Lustre: DEBUG MARKER: Tue Aug 1 13:30:46 2017 Aug 2 00:42:11 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 2 00:42:11 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 2 00:42:11 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 00:42:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 00:42:11 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 2 00:42:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 2 00:42:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa26c0 a5232eaa14f210cb Aug 2 00:42:11 oak-gw06 kernel: Call Trace: Aug 2 00:42:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 00:42:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 00:42:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 00:42:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 00:42:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 00:42:11 oak-gw06 kernel: [] ? __dev_kfree_skb_any+0x3d/0x50 Aug 2 00:42:11 oak-gw06 kernel: [] ? bnx2x_tx_int+0xc8/0x220 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 00:42:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 00:42:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 00:42:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 00:42:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 00:42:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 00:42:11 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 2 00:42:11 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 2 00:42:11 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 2 00:42:11 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 2 00:42:11 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 2 00:42:11 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 2 00:42:11 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 00:42:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 00:42:11 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 2 00:42:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 2 00:42:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa26c0 a5232eaa14f210cb Aug 2 00:42:11 oak-gw06 kernel: Call Trace: Aug 2 00:42:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 00:42:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 00:42:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 00:42:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 00:42:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 2 00:42:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 00:42:11 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 00:42:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 00:42:11 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 2 00:42:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 2 00:42:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa26c0 a5232eaa14f210cb Aug 2 00:42:11 oak-gw06 kernel: Call Trace: Aug 2 00:42:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 00:42:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 00:42:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 00:42:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 00:42:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 2 00:42:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 00:42:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 00:42:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 00:42:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 00:42:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 00:42:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 00:42:11 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 2 00:42:11 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 2 00:42:11 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 2 00:42:11 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 2 00:42:11 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 2 00:42:11 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 2 00:42:11 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 00:42:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 00:42:11 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 2 00:42:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 2 00:42:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa26c0 a5232eaa14f210cb Aug 2 00:42:11 oak-gw06 kernel: Call Trace: Aug 2 00:42:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 00:42:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 00:42:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 00:42:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 00:42:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 2 00:42:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 00:42:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 00:42:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 00:42:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 00:42:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 00:42:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 00:42:11 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 2 00:42:11 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 2 00:42:11 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 2 00:42:11 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 2 00:42:11 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 2 00:42:11 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 2 00:42:11 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 00:42:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 00:42:11 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 2 00:42:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 2 00:42:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa26c0 a5232eaa14f210cb Aug 2 00:42:11 oak-gw06 kernel: Call Trace: Aug 2 00:42:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 00:42:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 00:42:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 00:42:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 00:42:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 2 00:42:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 00:42:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 00:42:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 00:42:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 00:42:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 00:42:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 00:42:11 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 2 00:42:11 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 2 00:42:11 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 2 00:42:11 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 2 00:42:11 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 2 00:42:11 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 2 00:42:11 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 00:42:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 00:42:11 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 2 00:42:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 2 00:42:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa26c0 a5232eaa14f210cb Aug 2 00:42:11 oak-gw06 kernel: Call Trace: Aug 2 00:42:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 00:42:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 00:42:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 00:42:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 00:42:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 2 00:42:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 00:42:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 00:42:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 00:42:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 00:42:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 00:42:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 00:42:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 00:42:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 00:42:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 00:42:11 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 2 00:42:11 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 2 00:42:11 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 2 00:42:11 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 2 00:42:11 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 2 00:42:11 oak-gw06 kernel: swapper/3: page allocation failure: order:2, mode:0x104020 Aug 2 00:42:11 oak-gw06 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 00:42:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 00:42:11 oak-gw06 kernel: 0000000000104020 a5232eaa14f210cb ffff88043fcc39d8 ffffffff8168662f Aug 2 00:42:11 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 2 00:42:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa26c0 a5232eaa14f210cb Aug 2 00:42:11 oak-gw06 kernel: Call Trace: Aug 2 00:42:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 00:42:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 00:42:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 00:42:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 00:42:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 2 00:42:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 00:42:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 00:42:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 00:42:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 00:42:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 00:42:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 00:42:11 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 2 00:42:11 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 2 00:42:11 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 2 00:42:11 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 2 00:42:11 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 2 00:42:11 oak-gw06 kernel: CPU: 4 PID: 15645 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 00:42:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 00:42:11 oak-gw06 kernel: 0000000000104020 000000003020bbd3 ffff88043fd039d8 ffffffff8168662f Aug 2 00:42:11 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff880294244050 ffff880415b500b8 Aug 2 00:42:11 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88043fd03a18 000000003020bbd3 Aug 2 00:42:11 oak-gw06 kernel: Call Trace: Aug 2 00:42:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 00:42:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 00:42:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 00:42:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 00:42:11 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 00:42:11 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 00:42:11 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 00:42:11 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 00:42:11 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 00:42:11 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 00:42:11 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 00:42:11 oak-gw06 kernel: [] ? _raw_spin_lock+0x32/0x50 Aug 2 00:42:11 oak-gw06 kernel: [] osc_page_delete+0xf6/0x4e0 [osc] Aug 2 00:42:11 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 2 00:42:11 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 2 00:42:11 oak-gw06 kernel: [] ll_releasepage+0xee/0x1a0 [lustre] Aug 2 00:42:11 oak-gw06 kernel: [] try_to_release_page+0x32/0x50 Aug 2 00:42:11 oak-gw06 kernel: [] shrink_page_list+0x950/0xb00 Aug 2 00:42:11 oak-gw06 kernel: [] shrink_inactive_list+0x1fa/0x630 Aug 2 00:42:11 oak-gw06 kernel: [] shrink_lruvec+0x385/0x770 Aug 2 00:42:11 oak-gw06 kernel: [] ? wake_up_process+0x23/0x40 Aug 2 00:42:11 oak-gw06 kernel: [] shrink_zone+0x76/0x1a0 Aug 2 00:42:11 oak-gw06 kernel: [] do_try_to_free_pages+0xf0/0x4e0 Aug 2 00:42:11 oak-gw06 kernel: [] ? throttle_direct_reclaim+0xaa/0x2b0 Aug 2 00:42:11 oak-gw06 kernel: [] try_to_free_pages+0xfc/0x180 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x458/0x725 Aug 2 00:42:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 00:42:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 00:42:11 oak-gw06 kernel: [] __page_cache_alloc+0x97/0xb0 Aug 2 00:42:11 oak-gw06 kernel: [] grab_cache_page_nowait+0x2e/0xa0 Aug 2 00:42:11 oak-gw06 kernel: [] ll_write_begin+0xa1/0x830 [lustre] Aug 2 00:42:11 oak-gw06 kernel: [] generic_file_buffered_write+0x11e/0x2a0 Aug 2 00:42:11 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 2 00:42:11 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 2 00:42:11 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 2 00:42:11 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 2 00:42:11 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 2 00:42:11 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 2 00:42:11 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 2 00:42:11 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 2 00:42:11 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 2 00:42:11 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 2 00:42:11 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 2 00:55:22 oak-gw06 kernel: warn_alloc_failed: 695 callbacks suppressed Aug 2 00:55:22 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 2 00:55:22 oak-gw06 kernel: CPU: 6 PID: 15670 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 00:55:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 00:55:22 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 00:55:22 oak-gw06 kernel: 00000000000080d0 00000000748007b2 ffff880038c5b858 ffffffff8168662f Aug 2 00:55:22 oak-gw06 kernel: ffff880038c5b8e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 2 00:55:22 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff880038c5b8e8 00000000748007b2 Aug 2 00:55:22 oak-gw06 kernel: Call Trace: Aug 2 00:55:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 00:55:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 00:55:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 00:55:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 00:55:22 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 2 00:55:22 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 2 00:55:22 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 00:55:22 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 00:55:22 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 00:55:22 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 00:55:22 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 00:55:22 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 00:55:22 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 00:55:22 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 00:55:22 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 00:55:22 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 00:55:22 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 00:55:22 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 00:55:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 00:55:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 00:55:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 00:55:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 00:55:22 oak-gw06 kernel: Mem-Info: Aug 2 00:55:22 oak-gw06 kernel: active_anon:28813 inactive_anon:43567 isolated_anon:0#012 active_file:723805 inactive_file:1808789 isolated_file:0#012 unevictable:0 dirty:2686 writeback:3134 unstable:0#012 slab_reclaimable:35681 slab_unreclaimable:858235#012 mapped:6171 shmem:41048 pagetables:1654 bounce:0#012 free:432343 free_pcp:1484 free_cma:0 Aug 2 00:55:22 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 00:55:22 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 00:55:22 oak-gw06 kernel: Node 0 DMA32 free:314096kB min:11976kB low:14968kB high:17964kB active_anon:15448kB inactive_anon:31248kB active_file:505672kB inactive_file:1300948kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3348kB writeback:936kB mapped:2560kB shmem:31276kB slab_reclaimable:25844kB slab_unreclaimable:608284kB kernel_stack:992kB pagetables:1052kB unstable:0kB bounce:0kB free_pcp:1824kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 00:55:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 00:55:22 oak-gw06 kernel: Node 0 Normal free:1398344kB min:55536kB low:69420kB high:83304kB active_anon:100064kB inactive_anon:143020kB active_file:2389548kB inactive_file:5946084kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10628kB writeback:9832kB mapped:22124kB shmem:132916kB slab_reclaimable:116880kB slab_unreclaimable:2824640kB kernel_stack:4704kB pagetables:5564kB unstable:0kB bounce:0kB free_pcp:3164kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 00:55:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 00:55:22 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 00:55:22 oak-gw06 kernel: Node 0 DMA32: 2214*4kB (UE) 1816*8kB (UEM) 5180*16kB (UEM) 4813*32kB (UEM) 763*64kB (UEM) 29*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 313080kB Aug 2 00:55:22 oak-gw06 kernel: Node 0 Normal: 9020*4kB (UE) 5837*8kB (UEM) 23406*16kB (UEM) 22696*32kB (UEM) 3020*64kB (UEM) 170*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1398584kB Aug 2 00:55:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 00:55:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 00:55:22 oak-gw06 kernel: 2148216 total pagecache pages Aug 2 00:55:22 oak-gw06 kernel: 0 pages in swap cache Aug 2 00:55:22 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 00:55:22 oak-gw06 kernel: Free swap = 4194300kB Aug 2 00:55:22 oak-gw06 kernel: Total swap = 4194300kB Aug 2 00:55:22 oak-gw06 kernel: 4194203 pages RAM Aug 2 00:55:22 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 00:55:22 oak-gw06 kernel: 127313 pages reserved Aug 2 00:55:22 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 2 00:55:22 oak-gw06 kernel: CPU: 6 PID: 15670 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 00:55:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 00:55:22 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 00:55:22 oak-gw06 kernel: 00000000000080d0 00000000748007b2 ffff880038c5b808 ffffffff8168662f Aug 2 00:55:22 oak-gw06 kernel: ffff880038c5b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 00:55:22 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880038c5b868 00000000748007b2 Aug 2 00:55:22 oak-gw06 kernel: Call Trace: Aug 2 00:55:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 00:55:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 00:55:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 00:55:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 00:55:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 00:55:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 00:55:22 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 2 00:55:22 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 2 00:55:22 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 00:55:22 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 00:55:22 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 00:55:22 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 00:55:22 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 00:55:22 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 00:55:22 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 00:55:22 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 00:55:22 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 00:55:22 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 00:55:22 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 00:55:22 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 00:55:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 00:55:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 00:55:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 00:55:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 00:55:22 oak-gw06 kernel: Mem-Info: Aug 2 00:55:22 oak-gw06 kernel: active_anon:28813 inactive_anon:43567 isolated_anon:0#012 active_file:718344 inactive_file:1818394 isolated_file:0#012 unevictable:0 dirty:3061 writeback:2749 unstable:0#012 slab_reclaimable:35681 slab_unreclaimable:858231#012 mapped:6171 shmem:41048 pagetables:1654 bounce:0#012 free:435674 free_pcp:1604 free_cma:0 Aug 2 00:55:22 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 00:55:22 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 00:55:22 oak-gw06 kernel: Node 0 DMA32 free:314132kB min:11976kB low:14968kB high:17964kB active_anon:15448kB inactive_anon:31248kB active_file:502288kB inactive_file:1307244kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2692kB writeback:464kB mapped:2560kB shmem:31276kB slab_reclaimable:25844kB slab_unreclaimable:608348kB kernel_stack:992kB pagetables:1052kB unstable:0kB bounce:0kB free_pcp:3428kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 00:55:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 00:55:22 oak-gw06 kernel: Node 0 Normal free:1410780kB min:55536kB low:69420kB high:83304kB active_anon:100064kB inactive_anon:143020kB active_file:2359648kB inactive_file:5972544kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10328kB writeback:4928kB mapped:22124kB shmem:132916kB slab_reclaimable:116880kB slab_unreclaimable:2824560kB kernel_stack:4704kB pagetables:5564kB unstable:0kB bounce:0kB free_pcp:4152kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 00:55:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 00:55:22 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 00:55:22 oak-gw06 kernel: Node 0 DMA32: 2641*4kB (UEM) 1408*8kB (UEM) 5252*16kB (UEM) 4903*32kB (UEM) 786*64kB (UEM) 29*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 317028kB Aug 2 00:55:22 oak-gw06 kernel: Node 0 Normal: 8501*4kB (UEM) 5040*8kB (UEM) 23768*16kB (UEM) 23223*32kB (UEM) 3220*64kB (UEM) 195*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1429044kB Aug 2 00:55:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 00:55:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 00:55:22 oak-gw06 kernel: 2132134 total pagecache pages Aug 2 00:55:22 oak-gw06 kernel: 0 pages in swap cache Aug 2 00:55:22 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 00:55:22 oak-gw06 kernel: Free swap = 4194300kB Aug 2 00:55:22 oak-gw06 kernel: Total swap = 4194300kB Aug 2 00:55:22 oak-gw06 kernel: 4194203 pages RAM Aug 2 00:55:22 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 00:55:22 oak-gw06 kernel: 127313 pages reserved Aug 2 01:00:22 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 2 01:00:22 oak-gw06 kernel: CPU: 6 PID: 15656 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:00:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:00:22 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:00:22 oak-gw06 kernel: 00000000000080d0 00000000d38126a9 ffff880049dbb858 ffffffff8168662f Aug 2 01:00:22 oak-gw06 kernel: ffff880049dbb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:00:22 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880049dbb8b8 00000000d38126a9 Aug 2 01:00:22 oak-gw06 kernel: Call Trace: Aug 2 01:00:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:00:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:00:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:00:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:00:22 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 2 01:00:22 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 2 01:00:22 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:00:22 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:00:22 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:00:22 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:00:22 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:00:22 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:00:22 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:00:22 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:00:22 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:00:22 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:00:22 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:00:22 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:00:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:00:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:00:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:00:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:00:22 oak-gw06 kernel: Mem-Info: Aug 2 01:00:22 oak-gw06 kernel: active_anon:28830 inactive_anon:43567 isolated_anon:0#012 active_file:717337 inactive_file:1988531 isolated_file:0#012 unevictable:0 dirty:5276 writeback:936 unstable:0#012 slab_reclaimable:35523 slab_unreclaimable:843954#012 mapped:6182 shmem:41048 pagetables:1657 bounce:0#012 free:278436 free_pcp:697 free_cma:0 Aug 2 01:00:22 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:00:22 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:00:22 oak-gw06 kernel: Node 0 DMA32 free:162148kB min:11976kB low:14968kB high:17964kB active_anon:16096kB inactive_anon:31248kB active_file:511220kB inactive_file:1462732kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2532kB writeback:984kB mapped:2564kB shmem:31276kB slab_reclaimable:25700kB slab_unreclaimable:598048kB kernel_stack:992kB pagetables:1052kB unstable:0kB bounce:0kB free_pcp:1312kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:00:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:00:22 oak-gw06 kernel: Node 0 Normal free:928036kB min:55536kB low:69420kB high:83304kB active_anon:99224kB inactive_anon:143020kB active_file:2358128kB inactive_file:6497372kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:17840kB writeback:4096kB mapped:22164kB shmem:132916kB slab_reclaimable:116392kB slab_unreclaimable:2777752kB kernel_stack:4704kB pagetables:5576kB unstable:0kB bounce:0kB free_pcp:1660kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:00:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:00:22 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:00:22 oak-gw06 kernel: Node 0 DMA32: 424*4kB (UEM) 3596*8kB (UEM) 1325*16kB (UEM) 949*32kB (UEM) 969*64kB (UEM) 106*128kB (UM) 7*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 159408kB Aug 2 01:00:22 oak-gw06 kernel: Node 0 Normal: 1137*4kB (UEM) 12151*8kB (UEM) 7430*16kB (UEM) 11759*32kB (UEM) 4302*64kB (UEM) 341*128kB (UM) 18*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 920508kB Aug 2 01:00:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:00:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:00:22 oak-gw06 kernel: 2148013 total pagecache pages Aug 2 01:00:22 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:00:22 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:00:22 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:00:22 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:00:22 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:00:22 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:00:22 oak-gw06 kernel: 127313 pages reserved Aug 2 01:00:22 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 2 01:00:22 oak-gw06 kernel: CPU: 6 PID: 15656 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:00:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:00:22 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:00:22 oak-gw06 kernel: 00000000000080d0 00000000d38126a9 ffff880049dbb808 ffffffff8168662f Aug 2 01:00:22 oak-gw06 kernel: ffff880049dbb898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 2 01:00:22 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880049dbb868 00000000d38126a9 Aug 2 01:00:22 oak-gw06 kernel: Call Trace: Aug 2 01:00:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:00:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:00:22 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 2 01:00:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:00:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:00:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:00:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:00:22 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 2 01:00:22 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 2 01:00:22 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:00:22 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:00:22 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:00:22 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:00:22 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:00:22 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:00:22 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:00:22 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:00:22 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:00:22 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:00:22 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:00:22 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:00:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:00:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:00:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:00:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:00:22 oak-gw06 kernel: Mem-Info: Aug 2 01:00:22 oak-gw06 kernel: active_anon:28830 inactive_anon:43567 isolated_anon:0#012 active_file:717337 inactive_file:1995504 isolated_file:0#012 unevictable:0 dirty:5136 writeback:3010 unstable:0#012 slab_reclaimable:35523 slab_unreclaimable:844158#012 mapped:6182 shmem:41048 pagetables:1657 bounce:0#012 free:268229 free_pcp:691 free_cma:0 Aug 2 01:00:22 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:00:22 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:00:22 oak-gw06 kernel: Node 0 DMA32 free:153736kB min:11976kB low:14968kB high:17964kB active_anon:16096kB inactive_anon:31248kB active_file:511220kB inactive_file:1467684kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3384kB writeback:1252kB mapped:2564kB shmem:31276kB slab_reclaimable:25700kB slab_unreclaimable:598048kB kernel_stack:992kB pagetables:1052kB unstable:0kB bounce:0kB free_pcp:1460kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:00:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:00:22 oak-gw06 kernel: Node 0 Normal free:887716kB min:55536kB low:69420kB high:83304kB active_anon:99224kB inactive_anon:143020kB active_file:2358128kB inactive_file:6521032kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:17840kB writeback:8364kB mapped:22164kB shmem:132916kB slab_reclaimable:116392kB slab_unreclaimable:2778832kB kernel_stack:4704kB pagetables:5576kB unstable:0kB bounce:0kB free_pcp:1692kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:00:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:00:22 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:00:22 oak-gw06 kernel: Node 0 DMA32: 465*4kB (UEM) 3149*8kB (UEM) 1221*16kB (UEM) 949*32kB (UEM) 969*64kB (UEM) 106*128kB (UM) 7*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 154332kB Aug 2 01:00:22 oak-gw06 kernel: Node 0 Normal: 1217*4kB (UEM) 9304*8kB (UEM) 7214*16kB (UEM) 11469*32kB (UEM) 4303*64kB (UEM) 341*128kB (UM) 18*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 885380kB Aug 2 01:00:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:00:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:00:22 oak-gw06 kernel: 2141993 total pagecache pages Aug 2 01:00:22 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:00:22 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:00:22 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:00:22 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:00:22 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:00:22 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:00:22 oak-gw06 kernel: 127313 pages reserved Aug 2 01:05:22 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 2 01:05:22 oak-gw06 kernel: CPU: 6 PID: 15685 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:05:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:05:22 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:05:22 oak-gw06 kernel: 00000000000080d0 00000000d826ca1e ffff880237d5f858 ffffffff8168662f Aug 2 01:05:22 oak-gw06 kernel: ffff880237d5f8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:05:22 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880237d5f8b8 00000000d826ca1e Aug 2 01:05:22 oak-gw06 kernel: Call Trace: Aug 2 01:05:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:05:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:05:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:05:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:05:22 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 2 01:05:22 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 2 01:05:22 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:05:22 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:05:22 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:05:22 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:05:22 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:05:22 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:05:22 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:05:22 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:05:22 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:05:22 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:05:22 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:05:22 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:05:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:05:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:05:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:22 oak-gw06 kernel: Mem-Info: Aug 2 01:05:22 oak-gw06 kernel: active_anon:22908 inactive_anon:43567 isolated_anon:0#012 active_file:1062198 inactive_file:1712055 isolated_file:0#012 unevictable:0 dirty:4751 writeback:1315 unstable:0#012 slab_reclaimable:35535 slab_unreclaimable:845097#012 mapped:6202 shmem:41048 pagetables:1638 bounce:0#012 free:233923 free_pcp:1266 free_cma:0 Aug 2 01:05:22 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:05:22 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:05:22 oak-gw06 kernel: Node 0 DMA32 free:225468kB min:11976kB low:14968kB high:17964kB active_anon:15340kB inactive_anon:31248kB active_file:757284kB inactive_file:1161404kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3568kB writeback:204kB mapped:2588kB shmem:31276kB slab_reclaimable:25748kB slab_unreclaimable:597824kB kernel_stack:992kB pagetables:1112kB unstable:0kB bounce:0kB free_pcp:3180kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:05:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:05:22 oak-gw06 kernel: Node 0 Normal free:697232kB min:55536kB low:69420kB high:83304kB active_anon:77852kB inactive_anon:143020kB active_file:3491508kB inactive_file:5681096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:16168kB writeback:6996kB mapped:22220kB shmem:132916kB slab_reclaimable:116392kB slab_unreclaimable:2782548kB kernel_stack:4704kB pagetables:5440kB unstable:0kB bounce:0kB free_pcp:3336kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:05:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:05:22 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:05:22 oak-gw06 kernel: Node 0 DMA32: 1702*4kB (UEM) 2203*8kB (UEM) 1132*16kB (UEM) 2124*32kB (UEM) 1629*64kB (UEM) 103*128kB (UM) 5*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 229232kB Aug 2 01:05:22 oak-gw06 kernel: Node 0 Normal: 8739*4kB (UEM) 8688*8kB (UEM) 3463*16kB (UEM) 5170*32kB (UEM) 5222*64kB (UEM) 338*128kB (UM) 20*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 707900kB Aug 2 01:05:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:05:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:05:22 oak-gw06 kernel: 2144625 total pagecache pages Aug 2 01:05:22 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:05:22 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:05:22 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:05:22 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:05:22 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:05:22 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:05:22 oak-gw06 kernel: 127313 pages reserved Aug 2 01:05:22 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 2 01:05:22 oak-gw06 kernel: CPU: 0 PID: 15685 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:05:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:05:22 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:05:22 oak-gw06 kernel: 00000000000080d0 00000000d826ca1e ffff880237d5f808 ffffffff8168662f Aug 2 01:05:22 oak-gw06 kernel: ffff880237d5f898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:05:22 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880237d5f868 00000000d826ca1e Aug 2 01:05:22 oak-gw06 kernel: Call Trace: Aug 2 01:05:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:05:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:05:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:05:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:05:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:05:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:05:22 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 2 01:05:22 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 2 01:05:22 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:05:22 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:05:22 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:05:22 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:05:22 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:05:22 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:05:22 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:05:22 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:05:22 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:05:22 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:05:22 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:05:22 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:05:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:05:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:05:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:22 oak-gw06 kernel: Mem-Info: Aug 2 01:05:22 oak-gw06 kernel: active_anon:23168 inactive_anon:43567 isolated_anon:0#012 active_file:1062198 inactive_file:1708757 isolated_file:0#012 unevictable:0 dirty:4807 writeback:1190 unstable:0#012 slab_reclaimable:35535 slab_unreclaimable:845095#012 mapped:6202 shmem:41048 pagetables:1638 bounce:0#012 free:237711 free_pcp:838 free_cma:0 Aug 2 01:05:22 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:05:22 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:05:22 oak-gw06 kernel: Node 0 DMA32 free:228108kB min:11976kB low:14968kB high:17964kB active_anon:15340kB inactive_anon:31248kB active_file:757284kB inactive_file:1161548kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2672kB writeback:380kB mapped:2588kB shmem:31276kB slab_reclaimable:25748kB slab_unreclaimable:597824kB kernel_stack:992kB pagetables:1112kB unstable:0kB bounce:0kB free_pcp:1184kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:05:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:05:22 oak-gw06 kernel: Node 0 Normal free:697060kB min:55536kB low:69420kB high:83304kB active_anon:77592kB inactive_anon:143020kB active_file:3491508kB inactive_file:5680316kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:16944kB writeback:1952kB mapped:22220kB shmem:132916kB slab_reclaimable:116392kB slab_unreclaimable:2782540kB kernel_stack:4704kB pagetables:5440kB unstable:0kB bounce:0kB free_pcp:1360kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:05:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:05:22 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:05:22 oak-gw06 kernel: Node 0 DMA32: 1767*4kB (UE) 2011*8kB (UEM) 1012*16kB (UEM) 2125*32kB (UEM) 1630*64kB (UEM) 103*128kB (UM) 5*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 226132kB Aug 2 01:05:22 oak-gw06 kernel: Node 0 Normal: 7598*4kB (UEM) 8027*8kB (UEM) 3133*16kB (UEM) 5176*32kB (UEM) 5222*64kB (UEM) 338*128kB (UM) 20*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 692960kB Aug 2 01:05:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:05:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:05:22 oak-gw06 kernel: 2148389 total pagecache pages Aug 2 01:05:22 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:05:22 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:05:22 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:05:22 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:05:22 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:05:22 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:05:22 oak-gw06 kernel: 127313 pages reserved Aug 2 01:05:27 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 2 01:05:27 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 2 01:05:27 oak-gw06 kernel: CPU: 3 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:05:27 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:05:27 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fcc39d8 ffffffff8168662f Aug 2 01:05:27 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 2 01:05:27 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa7440 00000000da86253b Aug 2 01:05:27 oak-gw06 kernel: Call Trace: Aug 2 01:05:27 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:05:27 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:05:27 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:05:27 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:05:27 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 01:05:27 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 01:05:27 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 01:05:27 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 01:05:27 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 01:05:27 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_ap_completion.isra.30+0xda/0x4c0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_ap_completion.isra.30+0x306/0x4c0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] osc_extent_finish+0x3a2/0xb10 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_brw_fini_request+0xa72/0x12f0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] brw_interpret+0x332/0xe60 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 2 01:05:27 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:05:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:27 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:05:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:27 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 2 01:05:27 oak-gw06 kernel: CPU: 3 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:05:27 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:05:27 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fcc39d8 ffffffff8168662f Aug 2 01:05:27 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 2 01:05:27 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa7440 00000000da86253b Aug 2 01:05:27 oak-gw06 kernel: Call Trace: Aug 2 01:05:27 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:05:27 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:05:27 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:05:27 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:05:27 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 2 01:05:27 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 01:05:27 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 01:05:27 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 01:05:27 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 01:05:27 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 01:05:27 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_ap_completion.isra.30+0xda/0x4c0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_ap_completion.isra.30+0x306/0x4c0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] osc_extent_finish+0x3a2/0xb10 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_brw_fini_request+0xa72/0x12f0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] brw_interpret+0x332/0xe60 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 2 01:05:27 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:05:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:27 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:05:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:27 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 2 01:05:27 oak-gw06 kernel: CPU: 3 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:05:27 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:05:27 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fcc39d8 ffffffff8168662f Aug 2 01:05:27 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 2 01:05:27 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa64c0 00000000da86253b Aug 2 01:05:27 oak-gw06 kernel: Call Trace: Aug 2 01:05:27 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:05:27 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:05:27 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:05:27 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:05:27 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 01:05:27 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 01:05:27 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 01:05:27 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 01:05:27 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 01:05:27 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_ap_completion.isra.30+0xda/0x4c0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_ap_completion.isra.30+0x306/0x4c0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] osc_extent_finish+0x3a2/0xb10 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_brw_fini_request+0xa72/0x12f0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] brw_interpret+0x332/0xe60 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 2 01:05:27 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:05:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:27 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:05:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:27 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 2 01:05:27 oak-gw06 kernel: CPU: 3 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:05:27 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:05:27 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fcc39d8 ffffffff8168662f Aug 2 01:05:27 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 2 01:05:27 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa64c0 00000000da86253b Aug 2 01:05:27 oak-gw06 kernel: Call Trace: Aug 2 01:05:27 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:05:27 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:05:27 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:05:27 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:05:27 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 2 01:05:27 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 01:05:27 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 01:05:27 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 01:05:27 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 01:05:27 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 01:05:27 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_ap_completion.isra.30+0xda/0x4c0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_ap_completion.isra.30+0x306/0x4c0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] osc_extent_finish+0x3a2/0xb10 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_brw_fini_request+0xa72/0x12f0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] brw_interpret+0x332/0xe60 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 2 01:05:27 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:05:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:27 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:05:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:27 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 2 01:05:27 oak-gw06 kernel: CPU: 3 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:05:27 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:05:27 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fcc39d8 ffffffff8168662f Aug 2 01:05:27 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 2 01:05:27 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa64c0 00000000da86253b Aug 2 01:05:27 oak-gw06 kernel: Call Trace: Aug 2 01:05:27 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:05:27 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:05:27 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:05:27 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:05:27 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 2 01:05:27 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 01:05:27 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 01:05:27 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 01:05:27 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 01:05:27 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 01:05:27 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_ap_completion.isra.30+0xda/0x4c0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_ap_completion.isra.30+0x306/0x4c0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] osc_extent_finish+0x3a2/0xb10 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_brw_fini_request+0xa72/0x12f0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] brw_interpret+0x332/0xe60 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 2 01:05:27 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:05:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:27 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:05:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:27 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 2 01:05:27 oak-gw06 kernel: CPU: 3 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:05:27 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:05:27 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fcc39d8 ffffffff8168662f Aug 2 01:05:27 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 2 01:05:27 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa64c0 00000000da86253b Aug 2 01:05:27 oak-gw06 kernel: Call Trace: Aug 2 01:05:27 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:05:27 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:05:27 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:05:27 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:05:27 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 2 01:05:27 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 01:05:27 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 01:05:27 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 01:05:27 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 01:05:27 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 01:05:27 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_ap_completion.isra.30+0xda/0x4c0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_ap_completion.isra.30+0x306/0x4c0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] osc_extent_finish+0x3a2/0xb10 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_brw_fini_request+0xa72/0x12f0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] brw_interpret+0x332/0xe60 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 2 01:05:27 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:05:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:27 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:05:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:27 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 2 01:05:27 oak-gw06 kernel: CPU: 3 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:05:27 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:05:27 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fcc39d8 ffffffff8168662f Aug 2 01:05:27 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 2 01:05:27 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8801e0fa4d80 00000000da86253b Aug 2 01:05:27 oak-gw06 kernel: Call Trace: Aug 2 01:05:27 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:05:27 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:05:27 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:05:27 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:05:27 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 01:05:27 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 01:05:27 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 01:05:27 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 01:05:27 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 01:05:27 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_ap_completion.isra.30+0xda/0x4c0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_ap_completion.isra.30+0x306/0x4c0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] osc_extent_finish+0x3a2/0xb10 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? osc_brw_fini_request+0xa72/0x12f0 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] brw_interpret+0x332/0xe60 [osc] Aug 2 01:05:27 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 2 01:05:27 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 2 01:05:27 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:05:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:27 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:05:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:05:27 oak-gw06 kernel: CPU: 4 PID: 15722 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:05:27 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:05:27 oak-gw06 kernel: 0000000000104020 00000000a1c17e9b ffff88043fd039d8 ffffffff8168662f Aug 2 01:05:27 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 2 01:05:27 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff8803c223a6c0 00000000a1c17e9b Aug 2 01:05:27 oak-gw06 kernel: Call Trace: Aug 2 01:05:27 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:05:27 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:05:27 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:05:27 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:05:27 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:05:27 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 2 01:05:27 oak-gw06 kernel: [] ? __dev_kfree_skb_any+0x3d/0x50 Aug 2 01:05:27 oak-gw06 kernel: [] ? bnx2x_tx_int+0xc8/0x220 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 2 01:05:27 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 2 01:05:27 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 2 01:05:27 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 2 01:05:27 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 2 01:05:27 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 2 01:05:27 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 2 01:05:27 oak-gw06 kernel: [] ? cl_page_assume+0x77/0x200 [obdclass] Aug 2 01:05:27 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 2 01:05:27 oak-gw06 kernel: [] ? iov_iter_copy_from_user_atomic+0x6a/0xa0 Aug 2 01:05:27 oak-gw06 kernel: [] generic_file_buffered_write+0x155/0x2a0 Aug 2 01:05:27 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 2 01:05:27 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 2 01:05:27 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 2 01:05:27 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 2 01:05:27 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 2 01:05:27 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 2 01:05:27 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 2 01:05:27 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 2 01:05:27 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 2 01:05:27 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 2 01:05:27 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 2 01:10:22 oak-gw06 kernel: warn_alloc_failed: 240 callbacks suppressed Aug 2 01:10:22 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 2 01:10:22 oak-gw06 kernel: CPU: 6 PID: 15685 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:10:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:10:22 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:10:22 oak-gw06 kernel: 00000000000080d0 00000000d826ca1e ffff880237d5f858 ffffffff8168662f Aug 2 01:10:22 oak-gw06 kernel: ffff880237d5f8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:10:22 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880237d5f8b8 00000000d826ca1e Aug 2 01:10:22 oak-gw06 kernel: Call Trace: Aug 2 01:10:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:10:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:10:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:10:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:10:22 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 2 01:10:22 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 2 01:10:22 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:10:22 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:10:22 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:10:22 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:10:22 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:10:22 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:10:22 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:10:22 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:10:22 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:10:22 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:10:22 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:10:22 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:10:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:10:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:10:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:10:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:10:22 oak-gw06 kernel: Mem-Info: Aug 2 01:10:22 oak-gw06 kernel: active_anon:28847 inactive_anon:43567 isolated_anon:0#012 active_file:904060 inactive_file:1737851 isolated_file:0#012 unevictable:0 dirty:5935 writeback:1255 unstable:0#012 slab_reclaimable:35382 slab_unreclaimable:842669#012 mapped:6250 shmem:41048 pagetables:1660 bounce:0#012 free:338349 free_pcp:1681 free_cma:0 Aug 2 01:10:22 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:10:22 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:10:22 oak-gw06 kernel: Node 0 DMA32 free:219788kB min:11976kB low:14968kB high:17964kB active_anon:17784kB inactive_anon:31248kB active_file:654040kB inactive_file:1251556kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3680kB writeback:908kB mapped:2620kB shmem:31276kB slab_reclaimable:25612kB slab_unreclaimable:597136kB kernel_stack:1008kB pagetables:1116kB unstable:0kB bounce:0kB free_pcp:3348kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:10:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:10:22 oak-gw06 kernel: Node 0 Normal free:1119456kB min:55536kB low:69420kB high:83304kB active_anon:97604kB inactive_anon:143020kB active_file:2953532kB inactive_file:5709036kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:21956kB writeback:4500kB mapped:22380kB shmem:132916kB slab_reclaimable:115916kB slab_unreclaimable:2773524kB kernel_stack:4688kB pagetables:5524kB unstable:0kB bounce:0kB free_pcp:3764kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:10:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:10:22 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:10:22 oak-gw06 kernel: Node 0 DMA32: 2235*4kB (UEM) 2713*8kB (UEM) 179*16kB (UEM) 2909*32kB (UEM) 1256*64kB (UEM) 86*128kB (UM) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 219524kB Aug 2 01:10:22 oak-gw06 kernel: Node 0 Normal: 4300*4kB (UEM) 8270*8kB (UEM) 2193*16kB (UEM) 17521*32kB (UEM) 5946*64kB (UEM) 399*128kB (UM) 20*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1115856kB Aug 2 01:10:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:10:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:10:22 oak-gw06 kernel: 2133633 total pagecache pages Aug 2 01:10:22 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:10:22 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:10:22 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:10:22 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:10:22 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:10:22 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:10:22 oak-gw06 kernel: 127313 pages reserved Aug 2 01:10:22 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 2 01:10:22 oak-gw06 kernel: CPU: 6 PID: 15685 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:10:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:10:22 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:10:22 oak-gw06 kernel: 00000000000080d0 00000000d826ca1e ffff880237d5f808 ffffffff8168662f Aug 2 01:10:22 oak-gw06 kernel: ffff880237d5f898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:10:22 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880237d5f868 00000000d826ca1e Aug 2 01:10:22 oak-gw06 kernel: Call Trace: Aug 2 01:10:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:10:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:10:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:10:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:10:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:10:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:10:22 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 2 01:10:22 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 2 01:10:22 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:10:22 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:10:22 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:10:22 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:10:22 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:10:22 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:10:22 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:10:22 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:10:22 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:10:22 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:10:22 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:10:22 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:10:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:10:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:10:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:10:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:10:22 oak-gw06 kernel: Mem-Info: Aug 2 01:10:22 oak-gw06 kernel: active_anon:28847 inactive_anon:43567 isolated_anon:0#012 active_file:896624 inactive_file:1750757 isolated_file:0#012 unevictable:0 dirty:6034 writeback:2313 unstable:0#012 slab_reclaimable:35382 slab_unreclaimable:842669#012 mapped:6250 shmem:41048 pagetables:1660 bounce:0#012 free:334773 free_pcp:1047 free_cma:0 Aug 2 01:10:22 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:10:22 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:10:22 oak-gw06 kernel: Node 0 DMA32 free:217376kB min:11976kB low:14968kB high:17964kB active_anon:17784kB inactive_anon:31248kB active_file:651164kB inactive_file:1260020kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2956kB writeback:1476kB mapped:2620kB shmem:31276kB slab_reclaimable:25612kB slab_unreclaimable:597136kB kernel_stack:1008kB pagetables:1116kB unstable:0kB bounce:0kB free_pcp:1588kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:10:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:10:22 oak-gw06 kernel: Node 0 Normal free:1093076kB min:55536kB low:69420kB high:83304kB active_anon:97604kB inactive_anon:143020kB active_file:2935332kB inactive_file:5749336kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:20404kB writeback:2560kB mapped:22380kB shmem:132916kB slab_reclaimable:115916kB slab_unreclaimable:2773524kB kernel_stack:4688kB pagetables:5524kB unstable:0kB bounce:0kB free_pcp:3388kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:10:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:10:22 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:10:22 oak-gw06 kernel: Node 0 DMA32: 960*4kB (UEM) 2951*8kB (UEM) 180*16kB (UEM) 2890*32kB (UEM) 1256*64kB (UEM) 86*128kB (UM) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 215736kB Aug 2 01:10:22 oak-gw06 kernel: Node 0 Normal: 1224*4kB (UEM) 6912*8kB (UEM) 2085*16kB (UEM) 17521*32kB (UEM) 5946*64kB (UEM) 399*128kB (UM) 20*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1090960kB Aug 2 01:10:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:10:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:10:22 oak-gw06 kernel: 2141018 total pagecache pages Aug 2 01:10:22 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:10:22 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:10:22 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:10:22 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:10:22 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:10:22 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:10:22 oak-gw06 kernel: 127313 pages reserved Aug 2 01:15:22 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 2 01:15:22 oak-gw06 kernel: CPU: 0 PID: 15737 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:15:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:15:22 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:15:22 oak-gw06 kernel: 00000000000080d0 000000009ae0c83c ffff880365a6f858 ffffffff8168662f Aug 2 01:15:22 oak-gw06 kernel: ffff880365a6f8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:15:22 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880365a6f8b8 000000009ae0c83c Aug 2 01:15:22 oak-gw06 kernel: Call Trace: Aug 2 01:15:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:15:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:15:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:15:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:15:22 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 2 01:15:22 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 2 01:15:22 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:15:22 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:15:22 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:15:22 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:15:22 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:15:22 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:15:22 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:15:22 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:15:22 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:15:22 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:15:22 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:15:22 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:15:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:15:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:15:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:15:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:15:22 oak-gw06 kernel: Mem-Info: Aug 2 01:15:22 oak-gw06 kernel: active_anon:28640 inactive_anon:43567 isolated_anon:0#012 active_file:956058 inactive_file:1130474 isolated_file:0#012 unevictable:0 dirty:13266 writeback:1356 unstable:0#012 slab_reclaimable:35396 slab_unreclaimable:836279#012 mapped:6256 shmem:41048 pagetables:1639 bounce:0#012 free:925843 free_pcp:1917 free_cma:0 Aug 2 01:15:22 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:15:22 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:15:22 oak-gw06 kernel: Node 0 DMA32 free:753980kB min:11976kB low:14968kB high:17964kB active_anon:15160kB inactive_anon:31248kB active_file:615108kB inactive_file:785104kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:6904kB writeback:584kB mapped:2620kB shmem:31276kB slab_reclaimable:25644kB slab_unreclaimable:592684kB kernel_stack:992kB pagetables:1028kB unstable:0kB bounce:0kB free_pcp:3336kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:15:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:15:22 oak-gw06 kernel: Node 0 Normal free:2940508kB min:55536kB low:69420kB high:83304kB active_anon:99400kB inactive_anon:143020kB active_file:3216520kB inactive_file:3718448kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:45772kB writeback:8116kB mapped:22404kB shmem:132916kB slab_reclaimable:115940kB slab_unreclaimable:2752416kB kernel_stack:4704kB pagetables:5528kB unstable:0kB bounce:0kB free_pcp:3668kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:15:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:15:22 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:15:22 oak-gw06 kernel: Node 0 DMA32: 7473*4kB (UEM) 24436*8kB (UEM) 16315*16kB (UEM) 4173*32kB (UEM) 1839*64kB (UEM) 116*128kB (UM) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 754036kB Aug 2 01:15:22 oak-gw06 kernel: Node 0 Normal: 22128*4kB (UEM) 96550*8kB (UEM) 67688*16kB (UEM) 19206*32kB (UEM) 5148*64kB (UEM) 394*128kB (UM) 22*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2944048kB Aug 2 01:15:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:15:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:15:22 oak-gw06 kernel: 2124461 total pagecache pages Aug 2 01:15:22 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:15:22 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:15:22 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:15:22 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:15:22 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:15:22 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:15:22 oak-gw06 kernel: 127313 pages reserved Aug 2 01:15:22 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 2 01:15:22 oak-gw06 kernel: CPU: 0 PID: 15737 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:15:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:15:22 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:15:22 oak-gw06 kernel: 00000000000080d0 000000009ae0c83c ffff880365a6f808 ffffffff8168662f Aug 2 01:15:22 oak-gw06 kernel: ffff880365a6f898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:15:22 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880365a6f868 000000009ae0c83c Aug 2 01:15:22 oak-gw06 kernel: Call Trace: Aug 2 01:15:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:15:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:15:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:15:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:15:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:15:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:15:22 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 2 01:15:22 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 2 01:15:22 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:15:22 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:15:22 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:15:22 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:15:22 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:15:22 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:15:22 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:15:22 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:15:22 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:15:22 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:15:22 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:15:22 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:15:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:15:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:15:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:15:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:15:22 oak-gw06 kernel: Mem-Info: Aug 2 01:15:22 oak-gw06 kernel: active_anon:28640 inactive_anon:43567 isolated_anon:0#012 active_file:959944 inactive_file:1126711 isolated_file:0#012 unevictable:0 dirty:13697 writeback:2940 unstable:0#012 slab_reclaimable:35396 slab_unreclaimable:836279#012 mapped:6256 shmem:41048 pagetables:1639 bounce:0#012 free:927723 free_pcp:661 free_cma:0 Aug 2 01:15:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:15:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:15:23 oak-gw06 kernel: Node 0 DMA32 free:757412kB min:11976kB low:14968kB high:17964kB active_anon:15160kB inactive_anon:31248kB active_file:616236kB inactive_file:784352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:7464kB writeback:1704kB mapped:2620kB shmem:31276kB slab_reclaimable:25644kB slab_unreclaimable:592684kB kernel_stack:992kB pagetables:1028kB unstable:0kB bounce:0kB free_pcp:1656kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:15:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:15:23 oak-gw06 kernel: Node 0 Normal free:2936788kB min:55536kB low:69420kB high:83304kB active_anon:99400kB inactive_anon:143020kB active_file:3226920kB inactive_file:3725728kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:46160kB writeback:13936kB mapped:22404kB shmem:132916kB slab_reclaimable:115940kB slab_unreclaimable:2752416kB kernel_stack:4704kB pagetables:5528kB unstable:0kB bounce:0kB free_pcp:2528kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:15:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:15:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:15:23 oak-gw06 kernel: Node 0 DMA32: 7271*4kB (UEM) 24436*8kB (UEM) 16509*16kB (UEM) 4176*32kB (UEM) 1839*64kB (UEM) 116*128kB (UM) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 756428kB Aug 2 01:15:23 oak-gw06 kernel: Node 0 Normal: 18067*4kB (UEM) 96560*8kB (UEM) 67948*16kB (UEM) 19240*32kB (UEM) 5148*64kB (UEM) 394*128kB (UM) 22*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2933132kB Aug 2 01:15:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:15:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:15:23 oak-gw06 kernel: 2131024 total pagecache pages Aug 2 01:15:23 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:15:23 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:15:23 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:15:23 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:15:23 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:15:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:15:23 oak-gw06 kernel: 127313 pages reserved Aug 2 01:20:22 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 2 01:20:22 oak-gw06 kernel: CPU: 1 PID: 15737 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:20:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:20:22 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:20:22 oak-gw06 kernel: 00000000000080d0 000000009ae0c83c ffff880365a6f858 ffffffff8168662f Aug 2 01:20:22 oak-gw06 kernel: ffff880365a6f8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:20:22 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880365a6f8b8 000000009ae0c83c Aug 2 01:20:22 oak-gw06 kernel: Call Trace: Aug 2 01:20:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:20:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:20:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:20:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:20:22 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 2 01:20:22 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 2 01:20:22 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:20:22 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:20:22 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:20:22 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:20:22 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:20:22 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:20:22 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:20:22 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:20:22 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:20:22 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:20:22 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:20:22 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:20:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:20:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:20:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:20:22 oak-gw06 kernel: Mem-Info: Aug 2 01:20:22 oak-gw06 kernel: active_anon:22892 inactive_anon:43567 isolated_anon:0#012 active_file:1536402 inactive_file:543744 isolated_file:29#012 unevictable:0 dirty:13078 writeback:287 unstable:0#012 slab_reclaimable:36680 slab_unreclaimable:836139#012 mapped:6267 shmem:41048 pagetables:1618 bounce:0#012 free:953789 free_pcp:530 free_cma:0 Aug 2 01:20:22 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:20:22 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:20:22 oak-gw06 kernel: Node 0 DMA32 free:973312kB min:11976kB low:14968kB high:17964kB active_anon:13540kB inactive_anon:31248kB active_file:816156kB inactive_file:377624kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:11320kB writeback:0kB mapped:2620kB shmem:31276kB slab_reclaimable:26144kB slab_unreclaimable:589012kB kernel_stack:992kB pagetables:1024kB unstable:0kB bounce:0kB free_pcp:632kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:20:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:20:22 oak-gw06 kernel: Node 0 Normal free:2823032kB min:55536kB low:69420kB high:83304kB active_anon:78548kB inactive_anon:143020kB active_file:5329452kB inactive_file:1800212kB unevictable:0kB isolated(anon):0kB isolated(file):116kB present:13631488kB managed:13367060kB mlocked:0kB dirty:40992kB writeback:2048kB mapped:22448kB shmem:132916kB slab_reclaimable:120576kB slab_unreclaimable:2755528kB kernel_stack:4768kB pagetables:5448kB unstable:0kB bounce:0kB free_pcp:1356kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:20:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:20:22 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:20:22 oak-gw06 kernel: Node 0 DMA32: 3297*4kB (UE) 6650*8kB (UEM) 20311*16kB (UEM) 11731*32kB (UEM) 2825*64kB (UEM) 191*128kB (UM) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 973540kB Aug 2 01:20:22 oak-gw06 kernel: Node 0 Normal: 12970*4kB (UE) 72355*8kB (UEM) 69916*16kB (UEM) 22502*32kB (UEM) 4521*64kB (UEM) 409*128kB (UM) 24*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2817280kB Aug 2 01:20:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:20:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:20:22 oak-gw06 kernel: 2120049 total pagecache pages Aug 2 01:20:22 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:20:22 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:20:22 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:20:22 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:20:22 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:20:22 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:20:22 oak-gw06 kernel: 127313 pages reserved Aug 2 01:20:22 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 2 01:20:22 oak-gw06 kernel: CPU: 1 PID: 15737 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:20:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:20:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:20:23 oak-gw06 kernel: 00000000000080d0 000000009ae0c83c ffff880365a6f808 ffffffff8168662f Aug 2 01:20:23 oak-gw06 kernel: ffff880365a6f898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 2 01:20:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880365a6f868 000000009ae0c83c Aug 2 01:20:23 oak-gw06 kernel: Call Trace: Aug 2 01:20:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:20:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:20:23 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 2 01:20:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:20:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:20:23 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:20:23 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:20:23 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 2 01:20:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 2 01:20:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:20:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:20:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:20:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:20:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:20:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:20:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:20:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:20:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:20:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:20:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:20:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:20:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:20:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:20:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:20:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:20:23 oak-gw06 kernel: Mem-Info: Aug 2 01:20:23 oak-gw06 kernel: active_anon:22957 inactive_anon:43567 isolated_anon:0#012 active_file:1536467 inactive_file:549407 isolated_file:29#012 unevictable:0 dirty:13035 writeback:0 unstable:0#012 slab_reclaimable:36680 slab_unreclaimable:836139#012 mapped:6267 shmem:41048 pagetables:1618 bounce:0#012 free:947570 free_pcp:370 free_cma:0 Aug 2 01:20:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:20:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:20:23 oak-gw06 kernel: Node 0 DMA32 free:971232kB min:11976kB low:14968kB high:17964kB active_anon:13540kB inactive_anon:31248kB active_file:816156kB inactive_file:380256kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:10760kB writeback:48kB mapped:2620kB shmem:31276kB slab_reclaimable:26144kB slab_unreclaimable:589012kB kernel_stack:992kB pagetables:1024kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:20:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:20:23 oak-gw06 kernel: Node 0 Normal free:2801104kB min:55536kB low:69420kB high:83304kB active_anon:78548kB inactive_anon:143020kB active_file:5334392kB inactive_file:1813472kB unevictable:0kB isolated(anon):0kB isolated(file):116kB present:13631488kB managed:13367060kB mlocked:0kB dirty:40992kB writeback:884kB mapped:22448kB shmem:132916kB slab_reclaimable:120576kB slab_unreclaimable:2755528kB kernel_stack:4768kB pagetables:5448kB unstable:0kB bounce:0kB free_pcp:1804kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:20:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:20:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:20:23 oak-gw06 kernel: Node 0 DMA32: 3211*4kB (UEM) 6194*8kB (UEM) 20270*16kB (UEM) 11731*32kB (UEM) 2825*64kB (UEM) 191*128kB (UM) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 968892kB Aug 2 01:20:23 oak-gw06 kernel: Node 0 Normal: 12394*4kB (UE) 69869*8kB (UEM) 69784*16kB (UEM) 22501*32kB (UEM) 4521*64kB (UEM) 409*128kB (UM) 24*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2792944kB Aug 2 01:20:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:20:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:20:23 oak-gw06 kernel: 2124166 total pagecache pages Aug 2 01:20:23 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:20:23 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:20:23 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:20:23 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:20:23 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:20:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:20:23 oak-gw06 kernel: 127313 pages reserved Aug 2 01:25:22 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 2 01:25:22 oak-gw06 kernel: CPU: 6 PID: 15737 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:25:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:25:22 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:25:22 oak-gw06 kernel: 00000000000080d0 000000009ae0c83c ffff880365a6f858 ffffffff8168662f Aug 2 01:25:22 oak-gw06 kernel: ffff880365a6f8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:25:22 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880365a6f8b8 000000009ae0c83c Aug 2 01:25:22 oak-gw06 kernel: Call Trace: Aug 2 01:25:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:25:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:25:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:25:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:25:22 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 2 01:25:22 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 2 01:25:22 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:25:22 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:25:22 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:25:22 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:25:22 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:25:22 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:25:22 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:25:22 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:25:22 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:25:22 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:25:22 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:25:22 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:25:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:25:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:25:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:25:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:25:22 oak-gw06 kernel: Mem-Info: Aug 2 01:25:22 oak-gw06 kernel: active_anon:17469 inactive_anon:43567 isolated_anon:0#012 active_file:2024336 inactive_file:87289 isolated_file:0#012 unevictable:0 dirty:5900 writeback:0 unstable:0#012 slab_reclaimable:36542 slab_unreclaimable:842929#012 mapped:6212 shmem:41048 pagetables:1572 bounce:0#012 free:921368 free_pcp:161 free_cma:0 Aug 2 01:25:22 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:25:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:25:23 oak-gw06 kernel: Node 0 DMA32 free:1310128kB min:11976kB low:14968kB high:17964kB active_anon:13184kB inactive_anon:31248kB active_file:801172kB inactive_file:59864kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2272kB writeback:0kB mapped:2564kB shmem:31276kB slab_reclaimable:26036kB slab_unreclaimable:586632kB kernel_stack:976kB pagetables:1024kB unstable:0kB bounce:0kB free_pcp:372kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:25:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:25:23 oak-gw06 kernel: Node 0 Normal free:2359092kB min:55536kB low:69420kB high:83304kB active_anon:57212kB inactive_anon:143020kB active_file:7296172kB inactive_file:289292kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:21328kB writeback:0kB mapped:22284kB shmem:132916kB slab_reclaimable:120132kB slab_unreclaimable:2785068kB kernel_stack:4768kB pagetables:5264kB unstable:0kB bounce:0kB free_pcp:620kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:25:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:25:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:25:23 oak-gw06 kernel: Node 0 DMA32: 4131*4kB (UEM) 30633*8kB (UEM) 23826*16kB (UEM) 12837*32kB (UEM) 3449*64kB (UEM) 276*128kB (UM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1311700kB Aug 2 01:25:23 oak-gw06 kernel: Node 0 Normal: 16556*4kB (UEM) 49080*8kB (UEM) 63474*16kB (UEM) 17119*32kB (UEM) 4321*64kB (UEM) 427*128kB (UM) 24*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2359600kB Aug 2 01:25:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:25:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:25:23 oak-gw06 kernel: 2152673 total pagecache pages Aug 2 01:25:23 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:25:23 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:25:23 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:25:23 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:25:23 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:25:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:25:23 oak-gw06 kernel: 127313 pages reserved Aug 2 01:25:23 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 2 01:25:23 oak-gw06 kernel: CPU: 6 PID: 15737 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:25:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:25:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:25:23 oak-gw06 kernel: 00000000000080d0 000000009ae0c83c ffff880365a6f808 ffffffff8168662f Aug 2 01:25:23 oak-gw06 kernel: ffff880365a6f898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 2 01:25:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880365a6f868 000000009ae0c83c Aug 2 01:25:23 oak-gw06 kernel: Call Trace: Aug 2 01:25:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:25:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:25:23 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 2 01:25:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:25:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:25:23 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:25:23 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:25:23 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 2 01:25:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 2 01:25:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:25:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:25:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:25:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:25:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:25:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:25:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:25:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:25:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:25:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:25:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:25:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:25:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:25:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:25:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:25:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:25:23 oak-gw06 kernel: Mem-Info: Aug 2 01:25:23 oak-gw06 kernel: active_anon:17469 inactive_anon:43567 isolated_anon:0#012 active_file:2024336 inactive_file:87289 isolated_file:0#012 unevictable:0 dirty:5900 writeback:0 unstable:0#012 slab_reclaimable:36542 slab_unreclaimable:842929#012 mapped:6212 shmem:41048 pagetables:1572 bounce:0#012 free:921539 free_pcp:51 free_cma:0 Aug 2 01:25:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:25:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:25:23 oak-gw06 kernel: Node 0 DMA32 free:1310504kB min:11976kB low:14968kB high:17964kB active_anon:13184kB inactive_anon:31248kB active_file:801172kB inactive_file:59864kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2272kB writeback:0kB mapped:2564kB shmem:31276kB slab_reclaimable:26036kB slab_unreclaimable:586632kB kernel_stack:976kB pagetables:1024kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:25:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:25:23 oak-gw06 kernel: Node 0 Normal free:2359104kB min:55536kB low:69420kB high:83304kB active_anon:56692kB inactive_anon:143020kB active_file:7296172kB inactive_file:289292kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:21328kB writeback:0kB mapped:22284kB shmem:132916kB slab_reclaimable:120132kB slab_unreclaimable:2785068kB kernel_stack:4768kB pagetables:5264kB unstable:0kB bounce:0kB free_pcp:828kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:25:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:25:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:25:23 oak-gw06 kernel: Node 0 DMA32: 4222*4kB (UEM) 30632*8kB (UEM) 23827*16kB (UEM) 12838*32kB (UEM) 3449*64kB (UEM) 276*128kB (UM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1312104kB Aug 2 01:25:23 oak-gw06 kernel: Node 0 Normal: 16564*4kB (UEM) 49080*8kB (UEM) 63474*16kB (UEM) 17125*32kB (UEM) 4321*64kB (UEM) 427*128kB (UM) 24*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2359824kB Aug 2 01:25:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:25:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:25:23 oak-gw06 kernel: 2152545 total pagecache pages Aug 2 01:25:23 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:25:23 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:25:23 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:25:23 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:25:23 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:25:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:25:23 oak-gw06 kernel: 127313 pages reserved Aug 2 01:30:23 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 2 01:30:23 oak-gw06 kernel: CPU: 6 PID: 15934 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:30:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:30:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:30:23 oak-gw06 kernel: 00000000000080d0 000000005351ce67 ffff8801e17d7858 ffffffff8168662f Aug 2 01:30:23 oak-gw06 kernel: ffff8801e17d78e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:30:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801e17d78b8 000000005351ce67 Aug 2 01:30:23 oak-gw06 kernel: Call Trace: Aug 2 01:30:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:30:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:30:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:30:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:30:23 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 2 01:30:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 2 01:30:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:30:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:30:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:30:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:30:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:30:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:30:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:30:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:30:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:30:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:30:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:30:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:30:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:30:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:30:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:30:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:30:23 oak-gw06 kernel: Mem-Info: Aug 2 01:30:23 oak-gw06 kernel: active_anon:17153 inactive_anon:43567 isolated_anon:0#012 active_file:1836244 inactive_file:4063 isolated_file:0#012 unevictable:0 dirty:4509 writeback:170 unstable:0#012 slab_reclaimable:36550 slab_unreclaimable:851137#012 mapped:6128 shmem:41048 pagetables:1456 bounce:0#012 free:1185014 free_pcp:61 free_cma:0 Aug 2 01:30:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:30:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:30:23 oak-gw06 kernel: Node 0 DMA32 free:1454368kB min:11976kB low:14968kB high:17964kB active_anon:15172kB inactive_anon:31248kB active_file:722104kB inactive_file:1956kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1924kB writeback:292kB mapped:2504kB shmem:31276kB slab_reclaimable:26008kB slab_unreclaimable:588412kB kernel_stack:976kB pagetables:1024kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:30:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:30:23 oak-gw06 kernel: Node 0 Normal free:3269092kB min:55536kB low:69420kB high:83304kB active_anon:54220kB inactive_anon:143020kB active_file:6622872kB inactive_file:14296kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:16112kB writeback:388kB mapped:22008kB shmem:132916kB slab_reclaimable:120192kB slab_unreclaimable:2816120kB kernel_stack:4752kB pagetables:4800kB unstable:0kB bounce:0kB free_pcp:292kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:30:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:30:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:30:23 oak-gw06 kernel: Node 0 DMA32: 20331*4kB (UEM) 25708*8kB (UEM) 26486*16kB (UEM) 13667*32kB (UEM) 4005*64kB (UEM) 377*128kB (UM) 11*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1455500kB Aug 2 01:30:23 oak-gw06 kernel: Node 0 Normal: 125369*4kB (UEM) 124664*8kB (UEM) 59455*16kB (UEM) 14754*32kB (UEM) 4403*64kB (UEM) 463*128kB (UM) 24*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3269396kB Aug 2 01:30:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:30:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:30:23 oak-gw06 kernel: 1881326 total pagecache pages Aug 2 01:30:23 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:30:23 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:30:23 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:30:23 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:30:23 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:30:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:30:23 oak-gw06 kernel: 127313 pages reserved Aug 2 01:30:23 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 2 01:30:23 oak-gw06 kernel: CPU: 6 PID: 15934 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:30:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:30:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:30:23 oak-gw06 kernel: 00000000000080d0 000000005351ce67 ffff8801e17d7808 ffffffff8168662f Aug 2 01:30:23 oak-gw06 kernel: ffff8801e17d7898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:30:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801e17d7868 000000005351ce67 Aug 2 01:30:23 oak-gw06 kernel: Call Trace: Aug 2 01:30:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:30:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:30:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:30:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:30:23 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:30:23 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:30:23 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 2 01:30:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 2 01:30:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:30:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:30:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:30:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:30:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:30:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:30:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:30:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:30:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:30:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:30:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:30:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:30:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:30:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:30:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:30:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:30:23 oak-gw06 kernel: Mem-Info: Aug 2 01:30:23 oak-gw06 kernel: active_anon:17218 inactive_anon:43567 isolated_anon:0#012 active_file:1836114 inactive_file:4063 isolated_file:0#012 unevictable:0 dirty:4509 writeback:170 unstable:0#012 slab_reclaimable:36550 slab_unreclaimable:851137#012 mapped:6128 shmem:41048 pagetables:1456 bounce:0#012 free:1185062 free_pcp:84 free_cma:0 Aug 2 01:30:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:30:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:30:23 oak-gw06 kernel: Node 0 DMA32 free:1454368kB min:11976kB low:14968kB high:17964kB active_anon:15172kB inactive_anon:31248kB active_file:722104kB inactive_file:1956kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1924kB writeback:292kB mapped:2504kB shmem:31276kB slab_reclaimable:26008kB slab_unreclaimable:588412kB kernel_stack:976kB pagetables:1024kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:30:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:30:23 oak-gw06 kernel: Node 0 Normal free:3269988kB min:55536kB low:69420kB high:83304kB active_anon:53700kB inactive_anon:143020kB active_file:6622352kB inactive_file:14296kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:16112kB writeback:388kB mapped:22008kB shmem:132916kB slab_reclaimable:120192kB slab_unreclaimable:2816120kB kernel_stack:4752kB pagetables:4800kB unstable:0kB bounce:0kB free_pcp:328kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:30:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:30:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:30:23 oak-gw06 kernel: Node 0 DMA32: 20331*4kB (UEM) 25708*8kB (UEM) 26487*16kB (UEM) 13667*32kB (UEM) 4005*64kB (UEM) 377*128kB (UM) 11*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1455516kB Aug 2 01:30:23 oak-gw06 kernel: Node 0 Normal: 125331*4kB (UEM) 124688*8kB (UEM) 59458*16kB (UEM) 14755*32kB (UEM) 4403*64kB (UEM) 463*128kB (UM) 24*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3269516kB Aug 2 01:30:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:30:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:30:23 oak-gw06 kernel: 1881229 total pagecache pages Aug 2 01:30:23 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:30:23 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:30:23 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:30:23 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:30:23 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:30:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:30:23 oak-gw06 kernel: 127313 pages reserved Aug 2 01:35:23 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 2 01:35:23 oak-gw06 kernel: CPU: 6 PID: 15737 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:35:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:35:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:35:23 oak-gw06 kernel: 00000000000080d0 000000009ae0c83c ffff880365a6f858 ffffffff8168662f Aug 2 01:35:23 oak-gw06 kernel: ffff880365a6f8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:35:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880365a6f8b8 000000009ae0c83c Aug 2 01:35:23 oak-gw06 kernel: Call Trace: Aug 2 01:35:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:35:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:35:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:35:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:35:23 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 2 01:35:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 2 01:35:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:35:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:35:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:35:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:35:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:35:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:35:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:35:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:35:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:35:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:35:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:35:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:35:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:35:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:35:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:35:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:35:23 oak-gw06 kernel: Mem-Info: Aug 2 01:35:23 oak-gw06 kernel: active_anon:28384 inactive_anon:43567 isolated_anon:0#012 active_file:1895271 inactive_file:122394 isolated_file:0#012 unevictable:0 dirty:183 writeback:0 unstable:0#012 slab_reclaimable:36544 slab_unreclaimable:854186#012 mapped:6302 shmem:41048 pagetables:1654 bounce:0#012 free:993419 free_pcp:93 free_cma:0 Aug 2 01:35:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:35:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:35:23 oak-gw06 kernel: Node 0 DMA32 free:748500kB min:11976kB low:14968kB high:17964kB active_anon:18400kB inactive_anon:31248kB active_file:1321396kB inactive_file:87336kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:72kB writeback:0kB mapped:2652kB shmem:31276kB slab_reclaimable:26136kB slab_unreclaimable:594024kB kernel_stack:976kB pagetables:1032kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:35:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:35:23 oak-gw06 kernel: Node 0 Normal free:3208548kB min:55536kB low:69420kB high:83304kB active_anon:95656kB inactive_anon:143020kB active_file:6259688kB inactive_file:402240kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:660kB writeback:0kB mapped:22556kB shmem:132916kB slab_reclaimable:120040kB slab_unreclaimable:2822704kB kernel_stack:4720kB pagetables:5584kB unstable:0kB bounce:0kB free_pcp:456kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:35:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:35:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:35:23 oak-gw06 kernel: Node 0 DMA32: 3266*4kB (UEM) 7003*8kB (UEM) 2580*16kB (UEM) 10338*32kB (UEM) 4010*64kB (UEM) 387*128kB (UEM) 10*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 749920kB Aug 2 01:35:23 oak-gw06 kernel: Node 0 Normal: 43375*4kB (UEM) 88322*8kB (UEM) 73244*16kB (UEM) 23076*32kB (UEM) 5254*64kB (UEM) 588*128kB (UM) 28*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3209100kB Aug 2 01:35:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:35:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:35:23 oak-gw06 kernel: 2058517 total pagecache pages Aug 2 01:35:23 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:35:23 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:35:23 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:35:23 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:35:23 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:35:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:35:23 oak-gw06 kernel: 127313 pages reserved Aug 2 01:35:23 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 2 01:35:23 oak-gw06 kernel: CPU: 6 PID: 15737 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:35:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:35:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:35:23 oak-gw06 kernel: 00000000000080d0 000000009ae0c83c ffff880365a6f808 ffffffff8168662f Aug 2 01:35:23 oak-gw06 kernel: ffff880365a6f898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:35:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880365a6f868 000000009ae0c83c Aug 2 01:35:23 oak-gw06 kernel: Call Trace: Aug 2 01:35:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:35:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:35:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:35:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:35:23 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:35:23 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:35:23 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 2 01:35:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 2 01:35:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:35:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:35:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:35:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:35:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:35:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:35:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:35:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:35:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:35:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:35:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:35:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:35:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:35:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:35:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:35:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:35:23 oak-gw06 kernel: Mem-Info: Aug 2 01:35:23 oak-gw06 kernel: active_anon:28384 inactive_anon:43567 isolated_anon:0#012 active_file:1895151 inactive_file:122398 isolated_file:0#012 unevictable:0 dirty:180 writeback:0 unstable:0#012 slab_reclaimable:36544 slab_unreclaimable:854186#012 mapped:6305 shmem:41048 pagetables:1654 bounce:0#012 free:993983 free_pcp:31 free_cma:0 Aug 2 01:35:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:35:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:35:23 oak-gw06 kernel: Node 0 DMA32 free:750296kB min:11976kB low:14968kB high:17964kB active_anon:18400kB inactive_anon:31248kB active_file:1321012kB inactive_file:87384kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:72kB writeback:0kB mapped:2652kB shmem:31276kB slab_reclaimable:26136kB slab_unreclaimable:594024kB kernel_stack:976kB pagetables:1032kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:35:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:35:23 oak-gw06 kernel: Node 0 Normal free:3209100kB min:55536kB low:69420kB high:83304kB active_anon:95916kB inactive_anon:143020kB active_file:6259592kB inactive_file:402208kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:648kB writeback:0kB mapped:22568kB shmem:132916kB slab_reclaimable:120040kB slab_unreclaimable:2822704kB kernel_stack:4720kB pagetables:5584kB unstable:0kB bounce:0kB free_pcp:320kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:35:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:35:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:35:23 oak-gw06 kernel: Node 0 DMA32: 3267*4kB (UEM) 7004*8kB (UEM) 2606*16kB (UEM) 10339*32kB (UEM) 4010*64kB (UEM) 387*128kB (UEM) 10*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 750380kB Aug 2 01:35:23 oak-gw06 kernel: Node 0 Normal: 43438*4kB (UEM) 88328*8kB (UEM) 73244*16kB (UEM) 23076*32kB (UEM) 5254*64kB (UEM) 588*128kB (UM) 28*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3209400kB Aug 2 01:35:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:35:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:35:23 oak-gw06 kernel: 2058389 total pagecache pages Aug 2 01:35:23 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:35:23 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:35:23 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:35:23 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:35:23 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:35:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:35:23 oak-gw06 kernel: 127313 pages reserved Aug 2 01:40:23 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 2 01:40:23 oak-gw06 kernel: CPU: 6 PID: 16116 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:40:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:40:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:40:23 oak-gw06 kernel: 00000000000080d0 000000006de3b43e ffff88007ea0b858 ffffffff8168662f Aug 2 01:40:23 oak-gw06 kernel: ffff88007ea0b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:40:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88007ea0b8b8 000000006de3b43e Aug 2 01:40:23 oak-gw06 kernel: Call Trace: Aug 2 01:40:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:40:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:40:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:40:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:40:23 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 2 01:40:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 2 01:40:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:40:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:40:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:40:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:40:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:40:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:40:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:40:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:40:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:40:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:40:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:40:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:40:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:40:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:40:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:40:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:40:23 oak-gw06 kernel: Mem-Info: Aug 2 01:40:23 oak-gw06 kernel: active_anon:28840 inactive_anon:43567 isolated_anon:0#012 active_file:827721 inactive_file:1291592 isolated_file:0#012 unevictable:0 dirty:572 writeback:645 unstable:0#012 slab_reclaimable:36352 slab_unreclaimable:857495#012 mapped:6313 shmem:41048 pagetables:1662 bounce:0#012 free:887527 free_pcp:577 free_cma:0 Aug 2 01:40:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:40:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:40:23 oak-gw06 kernel: Node 0 DMA32 free:632356kB min:11976kB low:14968kB high:17964kB active_anon:18780kB inactive_anon:31248kB active_file:576120kB inactive_file:949592kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:996kB mapped:2652kB shmem:31276kB slab_reclaimable:26024kB slab_unreclaimable:597052kB kernel_stack:976kB pagetables:1032kB unstable:0kB bounce:0kB free_pcp:540kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:40:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:40:23 oak-gw06 kernel: Node 0 Normal free:2889992kB min:55536kB low:69420kB high:83304kB active_anon:96580kB inactive_anon:143020kB active_file:2734764kB inactive_file:4228708kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:1252kB writeback:1756kB mapped:22600kB shmem:132916kB slab_reclaimable:119384kB slab_unreclaimable:2832912kB kernel_stack:4704kB pagetables:5616kB unstable:0kB bounce:0kB free_pcp:2216kB local_pcp:4kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:40:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:40:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:40:23 oak-gw06 kernel: Node 0 DMA32: 3257*4kB (UEM) 6881*8kB (UEM) 4476*16kB (UEM) 5954*32kB (UEM) 3900*64kB (UEM) 378*128kB (UEM) 11*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 631020kB Aug 2 01:40:23 oak-gw06 kernel: Node 0 Normal: 8727*4kB (UEM) 62083*8kB (UEM) 64680*16kB (UEM) 26785*32kB (UEM) 5780*64kB (UEM) 658*128kB (UM) 33*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2886164kB Aug 2 01:40:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:40:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:40:23 oak-gw06 kernel: 2111125 total pagecache pages Aug 2 01:40:23 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:40:23 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:40:23 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:40:23 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:40:23 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:40:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:40:23 oak-gw06 kernel: 127313 pages reserved Aug 2 01:40:23 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 2 01:40:23 oak-gw06 kernel: CPU: 6 PID: 16116 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:40:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:40:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:40:23 oak-gw06 kernel: 00000000000080d0 000000006de3b43e ffff88007ea0b808 ffffffff8168662f Aug 2 01:40:23 oak-gw06 kernel: ffff88007ea0b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:40:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88007ea0b868 000000006de3b43e Aug 2 01:40:23 oak-gw06 kernel: Call Trace: Aug 2 01:40:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:40:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:40:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:40:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:40:23 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:40:23 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:40:23 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 2 01:40:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 2 01:40:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:40:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:40:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:40:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:40:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:40:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:40:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:40:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:40:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:40:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:40:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:40:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:40:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:40:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:40:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:40:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:40:23 oak-gw06 kernel: Mem-Info: Aug 2 01:40:23 oak-gw06 kernel: active_anon:28840 inactive_anon:43567 isolated_anon:0#012 active_file:827627 inactive_file:1300065 isolated_file:0#012 unevictable:0 dirty:335 writeback:796 unstable:0#012 slab_reclaimable:36352 slab_unreclaimable:857495#012 mapped:6313 shmem:41048 pagetables:1662 bounce:0#012 free:879352 free_pcp:687 free_cma:0 Aug 2 01:40:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:40:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:40:23 oak-gw06 kernel: Node 0 DMA32 free:626520kB min:11976kB low:14968kB high:17964kB active_anon:18780kB inactive_anon:31248kB active_file:575744kB inactive_file:955232kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:476kB writeback:436kB mapped:2652kB shmem:31276kB slab_reclaimable:26024kB slab_unreclaimable:597052kB kernel_stack:976kB pagetables:1032kB unstable:0kB bounce:0kB free_pcp:1160kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:40:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:40:23 oak-gw06 kernel: Node 0 Normal free:2862384kB min:55536kB low:69420kB high:83304kB active_anon:96580kB inactive_anon:143020kB active_file:2734764kB inactive_file:4257308kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:864kB writeback:4084kB mapped:22600kB shmem:132916kB slab_reclaimable:119384kB slab_unreclaimable:2832912kB kernel_stack:4704kB pagetables:5616kB unstable:0kB bounce:0kB free_pcp:2336kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:40:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:40:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:40:23 oak-gw06 kernel: Node 0 DMA32: 3027*4kB (UEM) 5968*8kB (UEM) 4489*16kB (UEM) 5954*32kB (UEM) 3900*64kB (UEM) 378*128kB (UEM) 11*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 623004kB Aug 2 01:40:23 oak-gw06 kernel: Node 0 Normal: 8720*4kB (UEM) 58460*8kB (UEM) 64672*16kB (UEM) 26785*32kB (UEM) 5780*64kB (UEM) 658*128kB (UM) 33*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2857024kB Aug 2 01:40:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:40:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:40:23 oak-gw06 kernel: 2120069 total pagecache pages Aug 2 01:40:23 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:40:23 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:40:23 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:40:23 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:40:23 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:40:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:40:23 oak-gw06 kernel: 127313 pages reserved Aug 2 01:45:23 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 2 01:45:23 oak-gw06 kernel: CPU: 3 PID: 15737 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:45:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:45:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:45:23 oak-gw06 kernel: 00000000000080d0 000000009ae0c83c ffff880365a6f858 ffffffff8168662f Aug 2 01:45:23 oak-gw06 kernel: ffff880365a6f8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:45:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880365a6f8b8 000000009ae0c83c Aug 2 01:45:23 oak-gw06 kernel: Call Trace: Aug 2 01:45:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:45:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:45:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:45:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:45:23 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 2 01:45:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 2 01:45:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:45:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:45:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:45:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:45:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:45:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:45:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:45:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:45:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:45:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:45:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:45:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:45:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:45:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:45:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:45:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:45:23 oak-gw06 kernel: Mem-Info: Aug 2 01:45:23 oak-gw06 kernel: active_anon:23882 inactive_anon:43567 isolated_anon:0#012 active_file:518907 inactive_file:1593358 isolated_file:0#012 unevictable:0 dirty:431 writeback:2351 unstable:0#012 slab_reclaimable:36184 slab_unreclaimable:856415#012 mapped:6325 shmem:41048 pagetables:1630 bounce:0#012 free:900368 free_pcp:1232 free_cma:0 Aug 2 01:45:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:45:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:45:23 oak-gw06 kernel: Node 0 DMA32 free:630836kB min:11976kB low:14968kB high:17964kB active_anon:17068kB inactive_anon:31248kB active_file:358436kB inactive_file:1168184kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:192kB writeback:1004kB mapped:2652kB shmem:31276kB slab_reclaimable:25896kB slab_unreclaimable:595912kB kernel_stack:976kB pagetables:1164kB unstable:0kB bounce:0kB free_pcp:2208kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:45:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:45:23 oak-gw06 kernel: Node 0 Normal free:2932032kB min:55536kB low:69420kB high:83304kB active_anon:78460kB inactive_anon:143020kB active_file:1717192kB inactive_file:5227352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:972kB writeback:1244kB mapped:22648kB shmem:132916kB slab_reclaimable:118840kB slab_unreclaimable:2829732kB kernel_stack:4704kB pagetables:5356kB unstable:0kB bounce:0kB free_pcp:3620kB local_pcp:100kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:45:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:45:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:45:23 oak-gw06 kernel: Node 0 DMA32: 2589*4kB (UEM) 2049*8kB (UEM) 4269*16kB (UEM) 7328*32kB (UEM) 3853*64kB (UEM) 386*128kB (UEM) 12*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 628620kB Aug 2 01:45:23 oak-gw06 kernel: Node 0 Normal: 8289*4kB (UE) 33805*8kB (UEM) 76938*16kB (UEM) 28360*32kB (UEM) 6059*64kB (UEM) 690*128kB (UM) 33*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2926668kB Aug 2 01:45:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:45:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:45:23 oak-gw06 kernel: 2114726 total pagecache pages Aug 2 01:45:23 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:45:23 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:45:23 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:45:23 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:45:23 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:45:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:45:23 oak-gw06 kernel: 127313 pages reserved Aug 2 01:45:23 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 2 01:45:23 oak-gw06 kernel: CPU: 7 PID: 15737 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:45:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:45:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:45:23 oak-gw06 kernel: 00000000000080d0 000000009ae0c83c ffff880365a6f808 ffffffff8168662f Aug 2 01:45:23 oak-gw06 kernel: ffff880365a6f898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:45:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880365a6f868 000000009ae0c83c Aug 2 01:45:23 oak-gw06 kernel: Call Trace: Aug 2 01:45:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:45:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:45:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:45:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:45:23 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:45:23 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:45:23 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 2 01:45:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 2 01:45:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:45:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:45:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:45:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:45:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:45:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:45:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:45:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:45:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:45:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:45:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:45:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:45:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:45:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:45:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:45:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:45:23 oak-gw06 kernel: Mem-Info: Aug 2 01:45:23 oak-gw06 kernel: active_anon:23882 inactive_anon:43567 isolated_anon:0#012 active_file:518907 inactive_file:1607748 isolated_file:0#012 unevictable:0 dirty:442 writeback:756 unstable:0#012 slab_reclaimable:36184 slab_unreclaimable:856415#012 mapped:6325 shmem:41048 pagetables:1630 bounce:0#012 free:886685 free_pcp:578 free_cma:0 Aug 2 01:45:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:45:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:45:23 oak-gw06 kernel: Node 0 DMA32 free:623336kB min:11976kB low:14968kB high:17964kB active_anon:17068kB inactive_anon:31248kB active_file:358436kB inactive_file:1177960kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1312kB writeback:1004kB mapped:2652kB shmem:31276kB slab_reclaimable:25896kB slab_unreclaimable:595912kB kernel_stack:976kB pagetables:1164kB unstable:0kB bounce:0kB free_pcp:852kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:45:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:45:23 oak-gw06 kernel: Node 0 Normal free:2897520kB min:55536kB low:69420kB high:83304kB active_anon:78460kB inactive_anon:143020kB active_file:1717192kB inactive_file:5260892kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:1748kB writeback:5124kB mapped:22648kB shmem:132916kB slab_reclaimable:118840kB slab_unreclaimable:2829732kB kernel_stack:4704kB pagetables:5356kB unstable:0kB bounce:0kB free_pcp:2976kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:45:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:45:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:45:23 oak-gw06 kernel: Node 0 DMA32: 3117*4kB (UEM) 2032*8kB (UE) 3513*16kB (UEM) 7329*32kB (UEM) 3853*64kB (UEM) 386*128kB (UEM) 12*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 618532kB Aug 2 01:45:23 oak-gw06 kernel: Node 0 Normal: 8392*4kB (UEM) 27981*8kB (UEM) 76812*16kB (UEM) 28364*32kB (UEM) 6059*64kB (UEM) 690*128kB (UM) 33*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2878600kB Aug 2 01:45:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:45:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:45:23 oak-gw06 kernel: 2129597 total pagecache pages Aug 2 01:45:23 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:45:23 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:45:23 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:45:23 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:45:23 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:45:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:45:23 oak-gw06 kernel: 127313 pages reserved Aug 2 01:50:23 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 2 01:50:23 oak-gw06 kernel: CPU: 6 PID: 16134 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:50:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:50:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:50:23 oak-gw06 kernel: 00000000000080d0 0000000000956a4d ffff88035ea17858 ffffffff8168662f Aug 2 01:50:23 oak-gw06 kernel: ffff88035ea178e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 2 01:50:23 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff88035ea178e8 0000000000956a4d Aug 2 01:50:23 oak-gw06 kernel: Call Trace: Aug 2 01:50:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:50:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:50:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:50:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:50:23 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 2 01:50:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 2 01:50:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:50:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:50:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:50:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:50:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:50:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:50:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:50:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:50:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:50:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:50:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:50:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:50:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:50:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:50:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:50:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:50:23 oak-gw06 kernel: Mem-Info: Aug 2 01:50:23 oak-gw06 kernel: active_anon:22129 inactive_anon:43567 isolated_anon:0#012 active_file:327201 inactive_file:1764807 isolated_file:17#012 unevictable:0 dirty:192 writeback:0 unstable:0#012 slab_reclaimable:36040 slab_unreclaimable:856944#012 mapped:6337 shmem:41048 pagetables:1607 bounce:0#012 free:923495 free_pcp:163 free_cma:0 Aug 2 01:50:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:50:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:50:23 oak-gw06 kernel: Node 0 DMA32 free:648024kB min:11976kB low:14968kB high:17964kB active_anon:18044kB inactive_anon:31248kB active_file:220448kB inactive_file:1290720kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:8kB writeback:0kB mapped:2652kB shmem:31276kB slab_reclaimable:25784kB slab_unreclaimable:596836kB kernel_stack:976kB pagetables:1168kB unstable:0kB bounce:0kB free_pcp:20kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:50:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:50:23 oak-gw06 kernel: Node 0 Normal free:3029692kB min:55536kB low:69420kB high:83304kB active_anon:70472kB inactive_anon:143020kB active_file:1088356kB inactive_file:5768508kB unevictable:0kB isolated(anon):0kB isolated(file):68kB present:13631488kB managed:13367060kB mlocked:0kB dirty:760kB writeback:0kB mapped:22696kB shmem:132916kB slab_reclaimable:118376kB slab_unreclaimable:2830924kB kernel_stack:4720kB pagetables:5260kB unstable:0kB bounce:0kB free_pcp:756kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:50:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:50:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:50:23 oak-gw06 kernel: Node 0 DMA32: 4272*4kB (UEM) 7041*8kB (UEM) 4835*16kB (UEM) 6247*32kB (UEM) 3806*64kB (UEM) 394*128kB (UEM) 13*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 648024kB Aug 2 01:50:23 oak-gw06 kernel: Node 0 Normal: 17077*4kB (UEM) 41345*8kB (UEM) 73317*16kB (UEM) 29944*32kB (UEM) 6262*64kB (UEM) 702*128kB (UM) 35*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3029932kB Aug 2 01:50:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:50:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:50:23 oak-gw06 kernel: 2133078 total pagecache pages Aug 2 01:50:23 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:50:23 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:50:23 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:50:23 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:50:23 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:50:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:50:23 oak-gw06 kernel: 127313 pages reserved Aug 2 01:50:23 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 2 01:50:23 oak-gw06 kernel: CPU: 6 PID: 16134 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 01:50:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 01:50:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 01:50:23 oak-gw06 kernel: 00000000000080d0 0000000000956a4d ffff88035ea17808 ffffffff8168662f Aug 2 01:50:23 oak-gw06 kernel: ffff88035ea17898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 01:50:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88035ea17868 0000000000956a4d Aug 2 01:50:23 oak-gw06 kernel: Call Trace: Aug 2 01:50:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 01:50:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 01:50:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 01:50:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 01:50:23 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 01:50:23 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 01:50:23 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 2 01:50:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 2 01:50:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 01:50:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 01:50:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 01:50:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 01:50:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 01:50:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 01:50:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 01:50:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 01:50:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 01:50:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 01:50:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 01:50:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 01:50:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 01:50:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:50:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 01:50:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 01:50:23 oak-gw06 kernel: Mem-Info: Aug 2 01:50:23 oak-gw06 kernel: active_anon:22129 inactive_anon:43567 isolated_anon:0#012 active_file:327201 inactive_file:1764742 isolated_file:17#012 unevictable:0 dirty:192 writeback:0 unstable:0#012 slab_reclaimable:36040 slab_unreclaimable:856944#012 mapped:6337 shmem:41048 pagetables:1607 bounce:0#012 free:923727 free_pcp:31 free_cma:0 Aug 2 01:50:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 01:50:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 01:50:23 oak-gw06 kernel: Node 0 DMA32 free:648024kB min:11976kB low:14968kB high:17964kB active_anon:18044kB inactive_anon:31248kB active_file:220448kB inactive_file:1290720kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:8kB writeback:0kB mapped:2652kB shmem:31276kB slab_reclaimable:25784kB slab_unreclaimable:596836kB kernel_stack:976kB pagetables:1168kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:50:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 01:50:23 oak-gw06 kernel: Node 0 Normal free:3030352kB min:55536kB low:69420kB high:83304kB active_anon:71252kB inactive_anon:143020kB active_file:1088356kB inactive_file:5768248kB unevictable:0kB isolated(anon):0kB isolated(file):68kB present:13631488kB managed:13367060kB mlocked:0kB dirty:760kB writeback:0kB mapped:22696kB shmem:132916kB slab_reclaimable:118376kB slab_unreclaimable:2830924kB kernel_stack:4720kB pagetables:5260kB unstable:0kB bounce:0kB free_pcp:276kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 01:50:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 01:50:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 01:50:23 oak-gw06 kernel: Node 0 DMA32: 4277*4kB (UEM) 7041*8kB (UEM) 4835*16kB (UEM) 6247*32kB (UEM) 3806*64kB (UEM) 394*128kB (UEM) 13*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 648044kB Aug 2 01:50:23 oak-gw06 kernel: Node 0 Normal: 17082*4kB (UEM) 41399*8kB (UEM) 73322*16kB (UEM) 29944*32kB (UEM) 6262*64kB (UEM) 702*128kB (UM) 35*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3030464kB Aug 2 01:50:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 01:50:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 01:50:23 oak-gw06 kernel: 2132981 total pagecache pages Aug 2 01:50:23 oak-gw06 kernel: 0 pages in swap cache Aug 2 01:50:23 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 01:50:23 oak-gw06 kernel: Free swap = 4194300kB Aug 2 01:50:23 oak-gw06 kernel: Total swap = 4194300kB Aug 2 01:50:23 oak-gw06 kernel: 4194203 pages RAM Aug 2 01:50:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 01:50:23 oak-gw06 kernel: 127313 pages reserved Aug 2 02:25:24 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 2 02:25:24 oak-gw06 kernel: CPU: 6 PID: 16203 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 02:25:24 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 02:25:24 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 02:25:24 oak-gw06 kernel: 00000000000080d0 0000000051078477 ffff88026aefb858 ffffffff8168662f Aug 2 02:25:24 oak-gw06 kernel: ffff88026aefb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 2 02:25:24 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88026aefb8b8 0000000051078477 Aug 2 02:25:24 oak-gw06 kernel: Call Trace: Aug 2 02:25:24 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 02:25:24 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 02:25:24 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 02:25:24 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 02:25:24 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 2 02:25:24 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 2 02:25:24 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 02:25:24 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 02:25:24 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 02:25:24 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 02:25:24 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 02:25:24 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 02:25:24 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 02:25:24 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 02:25:24 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 02:25:24 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 02:25:24 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 02:25:24 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 02:25:24 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 02:25:24 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 02:25:24 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 02:25:24 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 02:25:24 oak-gw06 kernel: Mem-Info: Aug 2 02:25:24 oak-gw06 kernel: active_anon:19442 inactive_anon:43567 isolated_anon:0#012 active_file:87335 inactive_file:1935628 isolated_file:0#012 unevictable:0 dirty:317 writeback:37 unstable:0#012 slab_reclaimable:35454 slab_unreclaimable:851175#012 mapped:6354 shmem:41048 pagetables:1432 bounce:0#012 free:1001874 free_pcp:159 free_cma:0 Aug 2 02:25:24 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 02:25:24 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 02:25:24 oak-gw06 kernel: Node 0 DMA32 free:666480kB min:11976kB low:14968kB high:17964kB active_anon:14196kB inactive_anon:31248kB active_file:52032kB inactive_file:1446404kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:320kB writeback:132kB mapped:2668kB shmem:31276kB slab_reclaimable:25508kB slab_unreclaimable:594864kB kernel_stack:976kB pagetables:1052kB unstable:0kB bounce:0kB free_pcp:516kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 02:25:24 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 02:25:24 oak-gw06 kernel: Node 0 Normal free:3324752kB min:55536kB low:69420kB high:83304kB active_anon:63572kB inactive_anon:143020kB active_file:297308kB inactive_file:6296108kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:948kB writeback:16kB mapped:22748kB shmem:132916kB slab_reclaimable:116308kB slab_unreclaimable:2809820kB kernel_stack:4720kB pagetables:4676kB unstable:0kB bounce:0kB free_pcp:268kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 02:25:24 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 02:25:24 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 02:25:24 oak-gw06 kernel: Node 0 DMA32: 9583*4kB (UEM) 12856*8kB (UEM) 4064*16kB (UEM) 4982*32kB (UEM) 3770*64kB (UEM) 449*128kB (UEM) 12*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 667452kB Aug 2 02:25:24 oak-gw06 kernel: Node 0 Normal: 51699*4kB (UEM) 63468*8kB (UEM) 42346*16kB (UEM) 37749*32kB (UEM) 8870*64kB (UEM) 1097*128kB (UM) 62*256kB (UM) 2*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 3325036kB Aug 2 02:25:24 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 02:25:24 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 02:25:24 oak-gw06 kernel: 2063825 total pagecache pages Aug 2 02:25:24 oak-gw06 kernel: 0 pages in swap cache Aug 2 02:25:24 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 02:25:24 oak-gw06 kernel: Free swap = 4194300kB Aug 2 02:25:24 oak-gw06 kernel: Total swap = 4194300kB Aug 2 02:25:24 oak-gw06 kernel: 4194203 pages RAM Aug 2 02:25:24 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 02:25:24 oak-gw06 kernel: 127313 pages reserved Aug 2 02:25:25 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 2 02:25:25 oak-gw06 kernel: CPU: 6 PID: 16203 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 2 02:25:25 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 2 02:25:25 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 2 02:25:25 oak-gw06 kernel: 00000000000080d0 0000000051078477 ffff88026aefb808 ffffffff8168662f Aug 2 02:25:25 oak-gw06 kernel: ffff88026aefb898 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 2 02:25:25 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff88026aefb898 0000000051078477 Aug 2 02:25:25 oak-gw06 kernel: Call Trace: Aug 2 02:25:25 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 2 02:25:25 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 2 02:25:25 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 2 02:25:25 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 2 02:25:25 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 2 02:25:25 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 2 02:25:25 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 2 02:25:25 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 2 02:25:25 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 2 02:25:25 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 2 02:25:25 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 2 02:25:25 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 2 02:25:25 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 2 02:25:25 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 2 02:25:25 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 2 02:25:25 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 2 02:25:25 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 2 02:25:25 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 2 02:25:25 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 2 02:25:25 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 2 02:25:25 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 2 02:25:25 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 02:25:25 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 2 02:25:25 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 2 02:25:25 oak-gw06 kernel: Mem-Info: Aug 2 02:25:25 oak-gw06 kernel: active_anon:19442 inactive_anon:43567 isolated_anon:0#012 active_file:87335 inactive_file:1938704 isolated_file:5#012 unevictable:0 dirty:252 writeback:316 unstable:0#012 slab_reclaimable:35454 slab_unreclaimable:850943#012 mapped:6361 shmem:41048 pagetables:1432 bounce:0#012 free:998557 free_pcp:359 free_cma:0 Aug 2 02:25:25 oak-gw06 kernel: Node 0 DMA free:15892kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 2 02:25:25 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 2 02:25:25 oak-gw06 kernel: Node 0 DMA32 free:664500kB min:11976kB low:14968kB high:17964kB active_anon:14196kB inactive_anon:31248kB active_file:52032kB inactive_file:1448756kB unevictable:0kB isolated(anon):0kB isolated(file):20kB present:3129332kB managed:2884592kB mlocked:0kB dirty:56kB writeback:876kB mapped:2676kB shmem:31276kB slab_reclaimable:25508kB slab_unreclaimable:594332kB kernel_stack:976kB pagetables:1052kB unstable:0kB bounce:0kB free_pcp:436kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 02:25:25 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 2 02:25:25 oak-gw06 kernel: Node 0 Normal free:3309400kB min:55536kB low:69420kB high:83304kB active_anon:63572kB inactive_anon:143020kB active_file:297308kB inactive_file:6307880kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:952kB writeback:1940kB mapped:22768kB shmem:132916kB slab_reclaimable:116308kB slab_unreclaimable:2809424kB kernel_stack:4720kB pagetables:4676kB unstable:0kB bounce:0kB free_pcp:1192kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 2 02:25:25 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 2 02:25:25 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 2 02:25:25 oak-gw06 kernel: Node 0 DMA32: 8187*4kB (UEM) 13007*8kB (UEM) 4196*16kB (UEM) 4990*32kB (UEM) 3738*64kB (UEM) 442*128kB (UEM) 12*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 662500kB Aug 2 02:25:25 oak-gw06 kernel: Node 0 Normal: 49014*4kB (UEM) 63806*8kB (UEM) 42358*16kB (UEM) 37579*32kB (UEM) 8834*64kB (UEM) 1097*128kB (UM) 62*256kB (UM) 2*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 3309448kB Aug 2 02:25:25 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 2 02:25:25 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 2 02:25:25 oak-gw06 kernel: 2068416 total pagecache pages Aug 2 02:25:25 oak-gw06 kernel: 0 pages in swap cache Aug 2 02:25:25 oak-gw06 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 2 02:25:25 oak-gw06 kernel: Free swap = 4194300kB Aug 2 02:25:25 oak-gw06 kernel: Total swap = 4194300kB Aug 2 02:25:25 oak-gw06 kernel: 4194203 pages RAM Aug 2 02:25:25 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 2 02:25:25 oak-gw06 kernel: 127313 pages reserved Aug 2 14:49:32 oak-gw06 kernel: Lustre: DEBUG MARKER: Wed Aug 2 14:49:32 2017 Aug 6 06:04:33 oak-gw06 kernel: INFO: task globus-gridftp-:29353 blocked for more than 120 seconds. Aug 6 06:04:33 oak-gw06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 6 06:04:33 oak-gw06 kernel: globus-gridftp- D ffff880280487e68 0 29353 1505 0x00000080 Aug 6 06:04:33 oak-gw06 kernel: ffff880280487a90 0000000000000082 ffff88042b548fb0 ffff880280487fd8 Aug 6 06:04:33 oak-gw06 kernel: ffff880280487fd8 ffff880280487fd8 ffff88042b548fb0 ffff8801515edd58 Aug 6 06:04:33 oak-gw06 kernel: ffff8801515edd60 7fffffffffffffff ffff88042b548fb0 ffff880280487e68 Aug 6 06:04:33 oak-gw06 kernel: Call Trace: Aug 6 06:04:33 oak-gw06 kernel: [] schedule+0x29/0x70 Aug 6 06:04:33 oak-gw06 kernel: [] schedule_timeout+0x239/0x2d0 Aug 6 06:04:33 oak-gw06 kernel: [] ? ptlrpc_set_add_new_req+0xe3/0x160 [ptlrpc] Aug 6 06:04:33 oak-gw06 kernel: [] ? osc_io_ladvise_end+0x50/0x50 [osc] Aug 6 06:04:33 oak-gw06 kernel: [] ? ptlrpcd_add_req+0x1ec/0x2e0 [ptlrpc] Aug 6 06:04:33 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 6 06:04:33 oak-gw06 kernel: [] wait_for_completion+0x116/0x170 Aug 6 06:04:33 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 6 06:04:33 oak-gw06 kernel: [] osc_io_setattr_end+0xc4/0x180 [osc] Aug 6 06:04:33 oak-gw06 kernel: [] ? lov_io_iter_fini_wrapper+0x50/0x50 [lov] Aug 6 06:04:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:04:33 oak-gw06 kernel: [] lov_io_end_wrapper+0xdb/0xe0 [lov] Aug 6 06:04:33 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 6 06:04:33 oak-gw06 kernel: [] lov_io_end+0x36/0xb0 [lov] Aug 6 06:04:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:04:33 oak-gw06 kernel: [] cl_io_loop+0xb3/0x190 [obdclass] Aug 6 06:04:33 oak-gw06 kernel: [] cl_setattr_ost+0x240/0x3a0 [lustre] Aug 6 06:04:33 oak-gw06 kernel: [] ll_setattr_raw+0x1299/0x1340 [lustre] Aug 6 06:04:33 oak-gw06 kernel: [] ? set_fd_set+0x21/0x30 Aug 6 06:04:33 oak-gw06 kernel: [] ll_setattr+0x63/0xc0 [lustre] Aug 6 06:04:33 oak-gw06 kernel: [] notify_change+0x279/0x3d0 Aug 6 06:04:33 oak-gw06 kernel: [] utimes_common+0xd9/0x200 Aug 6 06:04:33 oak-gw06 kernel: [] do_utimes+0xe5/0x180 Aug 6 06:04:33 oak-gw06 kernel: [] SyS_utime+0x6c/0xa0 Aug 6 06:04:33 oak-gw06 kernel: [] ? __audit_syscall_entry+0xb4/0x110 Aug 6 06:04:33 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 6 06:06:33 oak-gw06 kernel: INFO: task globus-gridftp-:29353 blocked for more than 120 seconds. Aug 6 06:06:33 oak-gw06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 6 06:06:33 oak-gw06 kernel: globus-gridftp- D ffff880280487e68 0 29353 1505 0x00000080 Aug 6 06:06:33 oak-gw06 kernel: ffff880280487a90 0000000000000082 ffff88042b548fb0 ffff880280487fd8 Aug 6 06:06:33 oak-gw06 kernel: ffff880280487fd8 ffff880280487fd8 ffff88042b548fb0 ffff8801515edd58 Aug 6 06:06:33 oak-gw06 kernel: ffff8801515edd60 7fffffffffffffff ffff88042b548fb0 ffff880280487e68 Aug 6 06:06:33 oak-gw06 kernel: Call Trace: Aug 6 06:06:33 oak-gw06 kernel: [] schedule+0x29/0x70 Aug 6 06:06:33 oak-gw06 kernel: [] schedule_timeout+0x239/0x2d0 Aug 6 06:06:33 oak-gw06 kernel: [] ? ptlrpc_set_add_new_req+0xe3/0x160 [ptlrpc] Aug 6 06:06:33 oak-gw06 kernel: [] ? osc_io_ladvise_end+0x50/0x50 [osc] Aug 6 06:06:33 oak-gw06 kernel: [] ? ptlrpcd_add_req+0x1ec/0x2e0 [ptlrpc] Aug 6 06:06:33 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 6 06:06:33 oak-gw06 kernel: [] wait_for_completion+0x116/0x170 Aug 6 06:06:33 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 6 06:06:33 oak-gw06 kernel: [] osc_io_setattr_end+0xc4/0x180 [osc] Aug 6 06:06:33 oak-gw06 kernel: [] ? lov_io_iter_fini_wrapper+0x50/0x50 [lov] Aug 6 06:06:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:06:33 oak-gw06 kernel: [] lov_io_end_wrapper+0xdb/0xe0 [lov] Aug 6 06:06:33 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 6 06:06:33 oak-gw06 kernel: [] lov_io_end+0x36/0xb0 [lov] Aug 6 06:06:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:06:33 oak-gw06 kernel: [] cl_io_loop+0xb3/0x190 [obdclass] Aug 6 06:06:33 oak-gw06 kernel: [] cl_setattr_ost+0x240/0x3a0 [lustre] Aug 6 06:06:33 oak-gw06 kernel: [] ll_setattr_raw+0x1299/0x1340 [lustre] Aug 6 06:06:33 oak-gw06 kernel: [] ? set_fd_set+0x21/0x30 Aug 6 06:06:33 oak-gw06 kernel: [] ll_setattr+0x63/0xc0 [lustre] Aug 6 06:06:33 oak-gw06 kernel: [] notify_change+0x279/0x3d0 Aug 6 06:06:33 oak-gw06 kernel: [] utimes_common+0xd9/0x200 Aug 6 06:06:33 oak-gw06 kernel: [] do_utimes+0xe5/0x180 Aug 6 06:06:33 oak-gw06 kernel: [] SyS_utime+0x6c/0xa0 Aug 6 06:06:33 oak-gw06 kernel: [] ? __audit_syscall_entry+0xb4/0x110 Aug 6 06:06:33 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 6 06:08:33 oak-gw06 kernel: INFO: task globus-gridftp-:29353 blocked for more than 120 seconds. Aug 6 06:08:33 oak-gw06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 6 06:08:33 oak-gw06 kernel: globus-gridftp- D ffff880280487e68 0 29353 1505 0x00000080 Aug 6 06:08:33 oak-gw06 kernel: ffff880280487a90 0000000000000082 ffff88042b548fb0 ffff880280487fd8 Aug 6 06:08:33 oak-gw06 kernel: ffff880280487fd8 ffff880280487fd8 ffff88042b548fb0 ffff8801515edd58 Aug 6 06:08:33 oak-gw06 kernel: ffff8801515edd60 7fffffffffffffff ffff88042b548fb0 ffff880280487e68 Aug 6 06:08:33 oak-gw06 kernel: Call Trace: Aug 6 06:08:33 oak-gw06 kernel: [] schedule+0x29/0x70 Aug 6 06:08:33 oak-gw06 kernel: [] schedule_timeout+0x239/0x2d0 Aug 6 06:08:33 oak-gw06 kernel: [] ? ptlrpc_set_add_new_req+0xe3/0x160 [ptlrpc] Aug 6 06:08:33 oak-gw06 kernel: [] ? osc_io_ladvise_end+0x50/0x50 [osc] Aug 6 06:08:33 oak-gw06 kernel: [] ? ptlrpcd_add_req+0x1ec/0x2e0 [ptlrpc] Aug 6 06:08:33 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 6 06:08:33 oak-gw06 kernel: [] wait_for_completion+0x116/0x170 Aug 6 06:08:33 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 6 06:08:33 oak-gw06 kernel: [] osc_io_setattr_end+0xc4/0x180 [osc] Aug 6 06:08:33 oak-gw06 kernel: [] ? lov_io_iter_fini_wrapper+0x50/0x50 [lov] Aug 6 06:08:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:08:33 oak-gw06 kernel: [] lov_io_end_wrapper+0xdb/0xe0 [lov] Aug 6 06:08:33 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 6 06:08:33 oak-gw06 kernel: [] lov_io_end+0x36/0xb0 [lov] Aug 6 06:08:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:08:33 oak-gw06 kernel: [] cl_io_loop+0xb3/0x190 [obdclass] Aug 6 06:08:33 oak-gw06 kernel: [] cl_setattr_ost+0x240/0x3a0 [lustre] Aug 6 06:08:33 oak-gw06 kernel: [] ll_setattr_raw+0x1299/0x1340 [lustre] Aug 6 06:08:33 oak-gw06 kernel: [] ? set_fd_set+0x21/0x30 Aug 6 06:08:33 oak-gw06 kernel: [] ll_setattr+0x63/0xc0 [lustre] Aug 6 06:08:33 oak-gw06 kernel: [] notify_change+0x279/0x3d0 Aug 6 06:08:33 oak-gw06 kernel: [] utimes_common+0xd9/0x200 Aug 6 06:08:33 oak-gw06 kernel: [] do_utimes+0xe5/0x180 Aug 6 06:08:33 oak-gw06 kernel: [] SyS_utime+0x6c/0xa0 Aug 6 06:08:33 oak-gw06 kernel: [] ? __audit_syscall_entry+0xb4/0x110 Aug 6 06:08:33 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 6 06:10:33 oak-gw06 kernel: INFO: task globus-gridftp-:29353 blocked for more than 120 seconds. Aug 6 06:10:33 oak-gw06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 6 06:10:33 oak-gw06 kernel: globus-gridftp- D ffff880280487e68 0 29353 1505 0x00000080 Aug 6 06:10:33 oak-gw06 kernel: ffff880280487a90 0000000000000082 ffff88042b548fb0 ffff880280487fd8 Aug 6 06:10:33 oak-gw06 kernel: ffff880280487fd8 ffff880280487fd8 ffff88042b548fb0 ffff8801515edd58 Aug 6 06:10:33 oak-gw06 kernel: ffff8801515edd60 7fffffffffffffff ffff88042b548fb0 ffff880280487e68 Aug 6 06:10:33 oak-gw06 kernel: Call Trace: Aug 6 06:10:33 oak-gw06 kernel: [] schedule+0x29/0x70 Aug 6 06:10:33 oak-gw06 kernel: [] schedule_timeout+0x239/0x2d0 Aug 6 06:10:33 oak-gw06 kernel: [] ? ptlrpc_set_add_new_req+0xe3/0x160 [ptlrpc] Aug 6 06:10:33 oak-gw06 kernel: [] ? osc_io_ladvise_end+0x50/0x50 [osc] Aug 6 06:10:33 oak-gw06 kernel: [] ? ptlrpcd_add_req+0x1ec/0x2e0 [ptlrpc] Aug 6 06:10:33 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 6 06:10:33 oak-gw06 kernel: [] wait_for_completion+0x116/0x170 Aug 6 06:10:33 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 6 06:10:33 oak-gw06 kernel: [] osc_io_setattr_end+0xc4/0x180 [osc] Aug 6 06:10:33 oak-gw06 kernel: [] ? lov_io_iter_fini_wrapper+0x50/0x50 [lov] Aug 6 06:10:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:10:33 oak-gw06 kernel: [] lov_io_end_wrapper+0xdb/0xe0 [lov] Aug 6 06:10:33 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 6 06:10:33 oak-gw06 kernel: [] lov_io_end+0x36/0xb0 [lov] Aug 6 06:10:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:10:33 oak-gw06 kernel: [] cl_io_loop+0xb3/0x190 [obdclass] Aug 6 06:10:33 oak-gw06 kernel: [] cl_setattr_ost+0x240/0x3a0 [lustre] Aug 6 06:10:33 oak-gw06 kernel: [] ll_setattr_raw+0x1299/0x1340 [lustre] Aug 6 06:10:33 oak-gw06 kernel: [] ? set_fd_set+0x21/0x30 Aug 6 06:10:33 oak-gw06 kernel: [] ll_setattr+0x63/0xc0 [lustre] Aug 6 06:10:33 oak-gw06 kernel: [] notify_change+0x279/0x3d0 Aug 6 06:10:33 oak-gw06 kernel: [] utimes_common+0xd9/0x200 Aug 6 06:10:33 oak-gw06 kernel: [] do_utimes+0xe5/0x180 Aug 6 06:10:33 oak-gw06 kernel: [] SyS_utime+0x6c/0xa0 Aug 6 06:10:33 oak-gw06 kernel: [] ? __audit_syscall_entry+0xb4/0x110 Aug 6 06:10:33 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 6 06:11:09 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502024468/real 1502024468] req@ffff880062d9e100 x1566268698945072/t0(0) o4->oak-OST001c-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/448 e 24 to 1 dl 1502025069 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 6 06:11:09 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Aug 6 06:11:09 oak-gw06 kernel: Lustre: oak-OST001c-osc-ffff88041b99c000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 06:11:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 6 06:11:09 oak-gw06 kernel: Lustre: oak-OST001c-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 6 06:11:11 oak-gw06 kernel: Lustre: oak-OST000e-osc-ffff88041b99c000: Connection to oak-OST000e (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 06:11:11 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 6 06:11:21 oak-gw06 kernel: Lustre: oak-OST0010-osc-ffff88041b99c000: Connection to oak-OST0010 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 06:11:26 oak-gw06 kernel: Lustre: oak-OST0000-osc-ffff88041b99c000: Connection to oak-OST0000 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 06:11:26 oak-gw06 kernel: Lustre: Skipped 4 previous similar messages Aug 6 06:12:33 oak-gw06 kernel: INFO: task globus-gridftp-:29353 blocked for more than 120 seconds. Aug 6 06:12:33 oak-gw06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 6 06:12:33 oak-gw06 kernel: globus-gridftp- D ffff880280487e68 0 29353 1505 0x00000080 Aug 6 06:12:33 oak-gw06 kernel: ffff880280487a90 0000000000000082 ffff88042b548fb0 ffff880280487fd8 Aug 6 06:12:33 oak-gw06 kernel: ffff880280487fd8 ffff880280487fd8 ffff88042b548fb0 ffff8801515edd58 Aug 6 06:12:33 oak-gw06 kernel: ffff8801515edd60 7fffffffffffffff ffff88042b548fb0 ffff880280487e68 Aug 6 06:12:33 oak-gw06 kernel: Call Trace: Aug 6 06:12:33 oak-gw06 kernel: [] schedule+0x29/0x70 Aug 6 06:12:33 oak-gw06 kernel: [] schedule_timeout+0x239/0x2d0 Aug 6 06:12:33 oak-gw06 kernel: [] ? ptlrpc_set_add_new_req+0xe3/0x160 [ptlrpc] Aug 6 06:12:33 oak-gw06 kernel: [] ? osc_io_ladvise_end+0x50/0x50 [osc] Aug 6 06:12:33 oak-gw06 kernel: [] ? ptlrpcd_add_req+0x1ec/0x2e0 [ptlrpc] Aug 6 06:12:33 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 6 06:12:33 oak-gw06 kernel: [] wait_for_completion+0x116/0x170 Aug 6 06:12:33 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 6 06:12:33 oak-gw06 kernel: [] osc_io_setattr_end+0xc4/0x180 [osc] Aug 6 06:12:33 oak-gw06 kernel: [] ? lov_io_iter_fini_wrapper+0x50/0x50 [lov] Aug 6 06:12:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:12:33 oak-gw06 kernel: [] lov_io_end_wrapper+0xdb/0xe0 [lov] Aug 6 06:12:33 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 6 06:12:33 oak-gw06 kernel: [] lov_io_end+0x36/0xb0 [lov] Aug 6 06:12:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:12:33 oak-gw06 kernel: [] cl_io_loop+0xb3/0x190 [obdclass] Aug 6 06:12:33 oak-gw06 kernel: [] cl_setattr_ost+0x240/0x3a0 [lustre] Aug 6 06:12:33 oak-gw06 kernel: [] ll_setattr_raw+0x1299/0x1340 [lustre] Aug 6 06:12:33 oak-gw06 kernel: [] ? set_fd_set+0x21/0x30 Aug 6 06:12:33 oak-gw06 kernel: [] ll_setattr+0x63/0xc0 [lustre] Aug 6 06:12:33 oak-gw06 kernel: [] notify_change+0x279/0x3d0 Aug 6 06:12:33 oak-gw06 kernel: [] utimes_common+0xd9/0x200 Aug 6 06:12:33 oak-gw06 kernel: [] do_utimes+0xe5/0x180 Aug 6 06:12:33 oak-gw06 kernel: [] SyS_utime+0x6c/0xa0 Aug 6 06:12:33 oak-gw06 kernel: [] ? __audit_syscall_entry+0xb4/0x110 Aug 6 06:12:33 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 6 06:14:33 oak-gw06 kernel: INFO: task globus-gridftp-:29353 blocked for more than 120 seconds. Aug 6 06:14:33 oak-gw06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 6 06:14:33 oak-gw06 kernel: globus-gridftp- D ffff880280487e68 0 29353 1505 0x00000080 Aug 6 06:14:33 oak-gw06 kernel: ffff880280487a90 0000000000000082 ffff88042b548fb0 ffff880280487fd8 Aug 6 06:14:33 oak-gw06 kernel: ffff880280487fd8 ffff880280487fd8 ffff88042b548fb0 ffff8801515edd58 Aug 6 06:14:33 oak-gw06 kernel: ffff8801515edd60 7fffffffffffffff ffff88042b548fb0 ffff880280487e68 Aug 6 06:14:33 oak-gw06 kernel: Call Trace: Aug 6 06:14:33 oak-gw06 kernel: [] schedule+0x29/0x70 Aug 6 06:14:33 oak-gw06 kernel: [] schedule_timeout+0x239/0x2d0 Aug 6 06:14:33 oak-gw06 kernel: [] ? ptlrpc_set_add_new_req+0xe3/0x160 [ptlrpc] Aug 6 06:14:33 oak-gw06 kernel: [] ? osc_io_ladvise_end+0x50/0x50 [osc] Aug 6 06:14:33 oak-gw06 kernel: [] ? ptlrpcd_add_req+0x1ec/0x2e0 [ptlrpc] Aug 6 06:14:33 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 6 06:14:33 oak-gw06 kernel: [] wait_for_completion+0x116/0x170 Aug 6 06:14:33 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 6 06:14:33 oak-gw06 kernel: [] osc_io_setattr_end+0xc4/0x180 [osc] Aug 6 06:14:33 oak-gw06 kernel: [] ? lov_io_iter_fini_wrapper+0x50/0x50 [lov] Aug 6 06:14:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:14:33 oak-gw06 kernel: [] lov_io_end_wrapper+0xdb/0xe0 [lov] Aug 6 06:14:33 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 6 06:14:33 oak-gw06 kernel: [] lov_io_end+0x36/0xb0 [lov] Aug 6 06:14:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:14:33 oak-gw06 kernel: [] cl_io_loop+0xb3/0x190 [obdclass] Aug 6 06:14:33 oak-gw06 kernel: [] cl_setattr_ost+0x240/0x3a0 [lustre] Aug 6 06:14:33 oak-gw06 kernel: [] ll_setattr_raw+0x1299/0x1340 [lustre] Aug 6 06:14:33 oak-gw06 kernel: [] ? set_fd_set+0x21/0x30 Aug 6 06:14:33 oak-gw06 kernel: [] ll_setattr+0x63/0xc0 [lustre] Aug 6 06:14:33 oak-gw06 kernel: [] notify_change+0x279/0x3d0 Aug 6 06:14:33 oak-gw06 kernel: [] utimes_common+0xd9/0x200 Aug 6 06:14:33 oak-gw06 kernel: [] do_utimes+0xe5/0x180 Aug 6 06:14:33 oak-gw06 kernel: [] SyS_utime+0x6c/0xa0 Aug 6 06:14:33 oak-gw06 kernel: [] ? __audit_syscall_entry+0xb4/0x110 Aug 6 06:14:33 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 6 06:16:33 oak-gw06 kernel: INFO: task globus-gridftp-:29353 blocked for more than 120 seconds. Aug 6 06:16:33 oak-gw06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 6 06:16:33 oak-gw06 kernel: globus-gridftp- D ffff880280487e68 0 29353 1505 0x00000080 Aug 6 06:16:33 oak-gw06 kernel: ffff880280487a90 0000000000000082 ffff88042b548fb0 ffff880280487fd8 Aug 6 06:16:33 oak-gw06 kernel: ffff880280487fd8 ffff880280487fd8 ffff88042b548fb0 ffff8801515edd58 Aug 6 06:16:33 oak-gw06 kernel: ffff8801515edd60 7fffffffffffffff ffff88042b548fb0 ffff880280487e68 Aug 6 06:16:33 oak-gw06 kernel: Call Trace: Aug 6 06:16:33 oak-gw06 kernel: [] schedule+0x29/0x70 Aug 6 06:16:33 oak-gw06 kernel: [] schedule_timeout+0x239/0x2d0 Aug 6 06:16:33 oak-gw06 kernel: [] ? ptlrpc_set_add_new_req+0xe3/0x160 [ptlrpc] Aug 6 06:16:33 oak-gw06 kernel: [] ? osc_io_ladvise_end+0x50/0x50 [osc] Aug 6 06:16:33 oak-gw06 kernel: [] ? ptlrpcd_add_req+0x1ec/0x2e0 [ptlrpc] Aug 6 06:16:33 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 6 06:16:33 oak-gw06 kernel: [] wait_for_completion+0x116/0x170 Aug 6 06:16:33 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 6 06:16:33 oak-gw06 kernel: [] osc_io_setattr_end+0xc4/0x180 [osc] Aug 6 06:16:33 oak-gw06 kernel: [] ? lov_io_iter_fini_wrapper+0x50/0x50 [lov] Aug 6 06:16:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:16:33 oak-gw06 kernel: [] lov_io_end_wrapper+0xdb/0xe0 [lov] Aug 6 06:16:33 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 6 06:16:33 oak-gw06 kernel: [] lov_io_end+0x36/0xb0 [lov] Aug 6 06:16:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:16:33 oak-gw06 kernel: [] cl_io_loop+0xb3/0x190 [obdclass] Aug 6 06:16:33 oak-gw06 kernel: [] cl_setattr_ost+0x240/0x3a0 [lustre] Aug 6 06:16:33 oak-gw06 kernel: [] ll_setattr_raw+0x1299/0x1340 [lustre] Aug 6 06:16:33 oak-gw06 kernel: [] ? set_fd_set+0x21/0x30 Aug 6 06:16:33 oak-gw06 kernel: [] ll_setattr+0x63/0xc0 [lustre] Aug 6 06:16:33 oak-gw06 kernel: [] notify_change+0x279/0x3d0 Aug 6 06:16:33 oak-gw06 kernel: [] utimes_common+0xd9/0x200 Aug 6 06:16:33 oak-gw06 kernel: [] do_utimes+0xe5/0x180 Aug 6 06:16:33 oak-gw06 kernel: [] SyS_utime+0x6c/0xa0 Aug 6 06:16:33 oak-gw06 kernel: [] ? __audit_syscall_entry+0xb4/0x110 Aug 6 06:16:33 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 6 06:18:33 oak-gw06 kernel: INFO: task globus-gridftp-:29353 blocked for more than 120 seconds. Aug 6 06:18:33 oak-gw06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 6 06:18:33 oak-gw06 kernel: globus-gridftp- D ffff880280487e68 0 29353 1505 0x00000080 Aug 6 06:18:33 oak-gw06 kernel: ffff880280487a90 0000000000000082 ffff88042b548fb0 ffff880280487fd8 Aug 6 06:18:33 oak-gw06 kernel: ffff880280487fd8 ffff880280487fd8 ffff88042b548fb0 ffff8801515edd58 Aug 6 06:18:33 oak-gw06 kernel: ffff8801515edd60 7fffffffffffffff ffff88042b548fb0 ffff880280487e68 Aug 6 06:18:33 oak-gw06 kernel: Call Trace: Aug 6 06:18:33 oak-gw06 kernel: [] schedule+0x29/0x70 Aug 6 06:18:33 oak-gw06 kernel: [] schedule_timeout+0x239/0x2d0 Aug 6 06:18:33 oak-gw06 kernel: [] ? ptlrpc_set_add_new_req+0xe3/0x160 [ptlrpc] Aug 6 06:18:33 oak-gw06 kernel: [] ? osc_io_ladvise_end+0x50/0x50 [osc] Aug 6 06:18:33 oak-gw06 kernel: [] ? ptlrpcd_add_req+0x1ec/0x2e0 [ptlrpc] Aug 6 06:18:33 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 6 06:18:33 oak-gw06 kernel: [] wait_for_completion+0x116/0x170 Aug 6 06:18:33 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 6 06:18:33 oak-gw06 kernel: [] osc_io_setattr_end+0xc4/0x180 [osc] Aug 6 06:18:33 oak-gw06 kernel: [] ? lov_io_iter_fini_wrapper+0x50/0x50 [lov] Aug 6 06:18:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:18:33 oak-gw06 kernel: [] lov_io_end_wrapper+0xdb/0xe0 [lov] Aug 6 06:18:33 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 6 06:18:33 oak-gw06 kernel: [] lov_io_end+0x36/0xb0 [lov] Aug 6 06:18:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:18:33 oak-gw06 kernel: [] cl_io_loop+0xb3/0x190 [obdclass] Aug 6 06:18:33 oak-gw06 kernel: [] cl_setattr_ost+0x240/0x3a0 [lustre] Aug 6 06:18:33 oak-gw06 kernel: [] ll_setattr_raw+0x1299/0x1340 [lustre] Aug 6 06:18:33 oak-gw06 kernel: [] ? set_fd_set+0x21/0x30 Aug 6 06:18:33 oak-gw06 kernel: [] ll_setattr+0x63/0xc0 [lustre] Aug 6 06:18:33 oak-gw06 kernel: [] notify_change+0x279/0x3d0 Aug 6 06:18:33 oak-gw06 kernel: [] utimes_common+0xd9/0x200 Aug 6 06:18:33 oak-gw06 kernel: [] do_utimes+0xe5/0x180 Aug 6 06:18:33 oak-gw06 kernel: [] SyS_utime+0x6c/0xa0 Aug 6 06:18:33 oak-gw06 kernel: [] ? __audit_syscall_entry+0xb4/0x110 Aug 6 06:18:33 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 6 06:18:33 oak-gw06 kernel: INFO: task globus-gridftp-:29406 blocked for more than 120 seconds. Aug 6 06:18:33 oak-gw06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 6 06:18:33 oak-gw06 kernel: globus-gridftp- D ffff880288673c90 0 29406 1505 0x00000080 Aug 6 06:18:33 oak-gw06 kernel: ffff8802886738c0 0000000000000086 ffff88000e3a2f10 ffff880288673fd8 Aug 6 06:18:33 oak-gw06 kernel: ffff880288673fd8 ffff880288673fd8 ffff88000e3a2f10 ffff8804006d2270 Aug 6 06:18:33 oak-gw06 kernel: ffff8804006d2278 7fffffffffffffff ffff88000e3a2f10 ffff880288673c90 Aug 6 06:18:33 oak-gw06 kernel: Call Trace: Aug 6 06:18:33 oak-gw06 kernel: [] schedule+0x29/0x70 Aug 6 06:18:33 oak-gw06 kernel: [] schedule_timeout+0x239/0x2d0 Aug 6 06:18:33 oak-gw06 kernel: [] ? ptlrpc_set_add_new_req+0xe3/0x160 [ptlrpc] Aug 6 06:18:33 oak-gw06 kernel: [] ? osc_io_ladvise_end+0x50/0x50 [osc] Aug 6 06:18:33 oak-gw06 kernel: [] ? ptlrpcd_add_req+0x1ec/0x2e0 [ptlrpc] Aug 6 06:18:33 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 6 06:18:33 oak-gw06 kernel: [] wait_for_completion+0x116/0x170 Aug 6 06:18:33 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 6 06:18:33 oak-gw06 kernel: [] osc_io_setattr_end+0xc4/0x180 [osc] Aug 6 06:18:33 oak-gw06 kernel: [] ? lov_io_iter_fini_wrapper+0x50/0x50 [lov] Aug 6 06:18:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:18:33 oak-gw06 kernel: [] lov_io_end_wrapper+0xdb/0xe0 [lov] Aug 6 06:18:33 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 6 06:18:33 oak-gw06 kernel: [] lov_io_end+0x36/0xb0 [lov] Aug 6 06:18:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:18:33 oak-gw06 kernel: [] cl_io_loop+0xb3/0x190 [obdclass] Aug 6 06:18:33 oak-gw06 kernel: [] cl_setattr_ost+0x240/0x3a0 [lustre] Aug 6 06:18:33 oak-gw06 kernel: [] ll_setattr_raw+0x1299/0x1340 [lustre] Aug 6 06:18:33 oak-gw06 kernel: [] ll_setattr+0x63/0xc0 [lustre] Aug 6 06:18:33 oak-gw06 kernel: [] notify_change+0x279/0x3d0 Aug 6 06:18:33 oak-gw06 kernel: [] ? security_inode_need_killpriv+0x16/0x20 Aug 6 06:18:33 oak-gw06 kernel: [] do_truncate+0x75/0xc0 Aug 6 06:18:33 oak-gw06 kernel: [] do_last+0x5f2/0x12a0 Aug 6 06:18:33 oak-gw06 kernel: [] path_openat+0xc2/0x490 Aug 6 06:18:33 oak-gw06 kernel: [] ? handle_mm_fault+0x6b1/0xfe0 Aug 6 06:18:33 oak-gw06 kernel: [] do_filp_open+0x4b/0xb0 Aug 6 06:18:33 oak-gw06 kernel: [] ? __alloc_fd+0xa7/0x130 Aug 6 06:18:33 oak-gw06 kernel: [] do_sys_open+0xf3/0x1f0 Aug 6 06:18:33 oak-gw06 kernel: [] SyS_open+0x1e/0x20 Aug 6 06:18:33 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 6 06:20:33 oak-gw06 kernel: INFO: task globus-gridftp-:29353 blocked for more than 120 seconds. Aug 6 06:20:33 oak-gw06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 6 06:20:33 oak-gw06 kernel: globus-gridftp- D ffff880280487e68 0 29353 1505 0x00000080 Aug 6 06:20:33 oak-gw06 kernel: ffff880280487a90 0000000000000082 ffff88042b548fb0 ffff880280487fd8 Aug 6 06:20:33 oak-gw06 kernel: ffff880280487fd8 ffff880280487fd8 ffff88042b548fb0 ffff8801515edd58 Aug 6 06:20:33 oak-gw06 kernel: ffff8801515edd60 7fffffffffffffff ffff88042b548fb0 ffff880280487e68 Aug 6 06:20:33 oak-gw06 kernel: Call Trace: Aug 6 06:20:33 oak-gw06 kernel: [] schedule+0x29/0x70 Aug 6 06:20:33 oak-gw06 kernel: [] schedule_timeout+0x239/0x2d0 Aug 6 06:20:33 oak-gw06 kernel: [] ? ptlrpc_set_add_new_req+0xe3/0x160 [ptlrpc] Aug 6 06:20:33 oak-gw06 kernel: [] ? osc_io_ladvise_end+0x50/0x50 [osc] Aug 6 06:20:33 oak-gw06 kernel: [] ? ptlrpcd_add_req+0x1ec/0x2e0 [ptlrpc] Aug 6 06:20:33 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 6 06:20:33 oak-gw06 kernel: [] wait_for_completion+0x116/0x170 Aug 6 06:20:33 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 6 06:20:33 oak-gw06 kernel: [] osc_io_setattr_end+0xc4/0x180 [osc] Aug 6 06:20:33 oak-gw06 kernel: [] ? lov_io_iter_fini_wrapper+0x50/0x50 [lov] Aug 6 06:20:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:20:33 oak-gw06 kernel: [] lov_io_end_wrapper+0xdb/0xe0 [lov] Aug 6 06:20:33 oak-gw06 kernel: [] lov_io_call.isra.9+0x86/0x140 [lov] Aug 6 06:20:33 oak-gw06 kernel: [] lov_io_end+0x36/0xb0 [lov] Aug 6 06:20:33 oak-gw06 kernel: [] cl_io_end+0x5d/0x150 [obdclass] Aug 6 06:20:33 oak-gw06 kernel: [] cl_io_loop+0xb3/0x190 [obdclass] Aug 6 06:20:33 oak-gw06 kernel: [] cl_setattr_ost+0x240/0x3a0 [lustre] Aug 6 06:20:33 oak-gw06 kernel: [] ll_setattr_raw+0x1299/0x1340 [lustre] Aug 6 06:20:33 oak-gw06 kernel: [] ? set_fd_set+0x21/0x30 Aug 6 06:20:33 oak-gw06 kernel: [] ll_setattr+0x63/0xc0 [lustre] Aug 6 06:20:33 oak-gw06 kernel: [] notify_change+0x279/0x3d0 Aug 6 06:20:33 oak-gw06 kernel: [] utimes_common+0xd9/0x200 Aug 6 06:20:33 oak-gw06 kernel: [] do_utimes+0xe5/0x180 Aug 6 06:20:33 oak-gw06 kernel: [] SyS_utime+0x6c/0xa0 Aug 6 06:20:33 oak-gw06 kernel: [] ? __audit_syscall_entry+0xb4/0x110 Aug 6 06:20:33 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 6 06:21:10 oak-gw06 kernel: Lustre: 1769:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502025069/real 1502025069] req@ffff88012b151800 x1566268698946128/t0(0) o4->oak-OST001c-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/448 e 18 to 1 dl 1502025670 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Aug 6 06:21:10 oak-gw06 kernel: Lustre: oak-OST001c-osc-ffff88041b99c000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 06:21:10 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 6 06:21:10 oak-gw06 kernel: Lustre: oak-OST001c-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 6 06:21:10 oak-gw06 kernel: Lustre: Skipped 9 previous similar messages Aug 6 06:21:10 oak-gw06 kernel: Lustre: 1769:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Aug 6 06:21:22 oak-gw06 kernel: Lustre: oak-OST0010-osc-ffff88041b99c000: Connection to oak-OST0010 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 06:21:22 oak-gw06 kernel: Lustre: Skipped 2 previous similar messages Aug 6 06:26:24 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502025383/real 1502025383] req@ffff88010d984300 x1566268698982624/t0(0) o10->oak-OST0026-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 560/432 e 1 to 1 dl 1502025984 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Aug 6 06:26:24 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Aug 6 06:26:24 oak-gw06 kernel: Lustre: oak-OST0026-osc-ffff88041b99c000: Connection to oak-OST0026 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 06:26:24 oak-gw06 kernel: Lustre: Skipped 6 previous similar messages Aug 6 06:26:24 oak-gw06 kernel: Lustre: oak-OST0026-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 6 06:26:24 oak-gw06 kernel: Lustre: Skipped 9 previous similar messages Aug 6 06:31:11 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502025670/real 1502025670] req@ffff88016a198600 x1566268698945296/t0(0) o2->oak-OST001c-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 560/432 e 23 to 1 dl 1502026271 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Aug 6 06:31:11 oak-gw06 kernel: Lustre: oak-OST001c-osc-ffff88041b99c000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 06:31:11 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Aug 6 06:31:28 oak-gw06 kernel: Lustre: oak-OST0004-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 6 06:31:28 oak-gw06 kernel: Lustre: Skipped 4 previous similar messages Aug 6 06:36:25 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502025984/real 1502025984] req@ffff88010d984300 x1566268698982624/t0(0) o10->oak-OST0026-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 560/432 e 1 to 1 dl 1502026585 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Aug 6 06:36:25 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Aug 6 06:36:25 oak-gw06 kernel: Lustre: oak-OST0026-osc-ffff88041b99c000: Connection to oak-OST0026 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 06:36:25 oak-gw06 kernel: Lustre: Skipped 9 previous similar messages Aug 6 06:41:12 oak-gw06 kernel: Lustre: oak-OST001c-osc-ffff88041b99c000: Connection to oak-OST001c (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 06:41:29 oak-gw06 kernel: Lustre: oak-OST001a-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 6 06:41:29 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Aug 6 06:46:26 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502026585/real 1502026585] req@ffff88010d984300 x1566268698982624/t0(0) o10->oak-OST0026-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 560/432 e 1 to 1 dl 1502027186 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Aug 6 06:46:26 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Aug 6 06:46:26 oak-gw06 kernel: Lustre: oak-OST0026-osc-ffff88041b99c000: Connection to oak-OST0026 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 06:46:26 oak-gw06 kernel: Lustre: Skipped 9 previous similar messages Aug 6 06:51:30 oak-gw06 kernel: Lustre: oak-OST0022-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 6 06:51:30 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Aug 6 06:56:27 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502027186/real 1502027186] req@ffff88010d984300 x1566268698982624/t0(0) o10->oak-OST0026-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 560/432 e 1 to 1 dl 1502027787 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Aug 6 06:56:27 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Aug 6 06:56:27 oak-gw06 kernel: Lustre: oak-OST0026-osc-ffff88041b99c000: Connection to oak-OST0026 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 06:56:27 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Aug 6 07:01:31 oak-gw06 kernel: Lustre: oak-OST001a-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 6 07:01:31 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Aug 6 07:06:28 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502027787/real 1502027787] req@ffff88010d984300 x1566268698982624/t0(0) o10->oak-OST0026-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 560/432 e 1 to 1 dl 1502028388 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Aug 6 07:06:28 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Aug 6 07:06:28 oak-gw06 kernel: Lustre: oak-OST0026-osc-ffff88041b99c000: Connection to oak-OST0026 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 07:06:28 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Aug 6 07:11:32 oak-gw06 kernel: Lustre: oak-OST001a-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 6 07:11:32 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Aug 6 07:16:19 oak-gw06 kernel: LustreError: 2073:0:(osc_cache.c:947:osc_extent_wait()) extent ffff88026ba8fdc8@{[0 -> 64/255], [3|0|+|rpc|wiuY|ffff88042129b1d0], [290816|65|+|-|ffff88001439f600|256|ffff88042c4cde20]} oak-OST0012-osc-ffff88041b99c000: wait ext to 0 timedout, recovery in progress? Aug 6 07:16:19 oak-gw06 kernel: LustreError: 1985:0:(osc_cache.c:947:osc_extent_wait()) extent ffff88003947e348@{[0 -> 0/255], [3|0|+|rpc|wiuY|ffff88042129af70], [28672|1|+|-|ffff88001439da00|256|ffff88042c4cde20]} oak-OST000e-osc-ffff88041b99c000: wait ext to 0 timedout, recovery in progress? Aug 6 07:16:19 oak-gw06 kernel: LustreError: 1985:0:(osc_cache.c:947:osc_extent_wait()) ### extent: ffff88003947e348 ns: oak-OST000e-osc-ffff88041b99c000 lock: ffff88001439da00/0xf077f1a82c38ca31 lrc: 3/0,0 mode: PW/PW res: [0x31af51:0x0:0x0].0x0 rrc: 1 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x29400000000 nid: local remote: 0xd7f0509f5e5f02e8 expref: -99 pid: 29353 timeout: 0 lvb_type: 1 Aug 6 07:16:29 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502028388/real 1502028388] req@ffff88010d984300 x1566268698982624/t0(0) o10->oak-OST0026-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 560/432 e 1 to 1 dl 1502028989 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Aug 6 07:16:29 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Aug 6 07:16:29 oak-gw06 kernel: Lustre: oak-OST0026-osc-ffff88041b99c000: Connection to oak-OST0026 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 07:16:29 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Aug 6 07:16:29 oak-gw06 kernel: LustreError: 2076:0:(osc_cache.c:947:osc_extent_wait()) extent ffff880178098738@{[0 -> 64/255], [3|0|+|rpc|wiuY|ffff88042129bdb0], [290816|65|+|-|ffff88010b1e9400|256|ffff88042c4cde20]} oak-OST0024-osc-ffff88041b99c000: wait ext to 0 timedout, recovery in progress? Aug 6 07:16:29 oak-gw06 kernel: LustreError: 2091:0:(osc_cache.c:947:osc_extent_wait()) extent ffff8801c2e52498@{[0 -> 0/255], [3|0|+|rpc|wiuY|ffff88042129b690], [28672|1|+|-|ffff880216540600|256|ffff88042c4cde20]} oak-OST0022-osc-ffff88041b99c000: wait ext to 0 timedout, recovery in progress? Aug 6 07:16:29 oak-gw06 kernel: LustreError: 2036:0:(osc_cache.c:947:osc_extent_wait()) extent ffff880014497b28@{[0 -> 0/255], [3|0|+|rpc|wiuY|ffff8802ff710000], [28672|1|+|-|ffff880400354200|256|ffff88042c4cde20]} oak-OST0006-osc-ffff88041b99c000: wait ext to 0 timedout, recovery in progress? Aug 6 07:16:29 oak-gw06 kernel: LustreError: 2044:0:(osc_cache.c:947:osc_extent_wait()) extent ffff880178098e70@{[0 -> 64/255], [3|0|+|rpc|wiuY|ffff880161465560], [290816|65|+|-|ffff880216541400|256|ffff88042c4cde20]} oak-OST0004-osc-ffff88041b99c000: wait ext to 0 timedout, recovery in progress? Aug 6 07:16:29 oak-gw06 kernel: LustreError: 2091:0:(osc_cache.c:947:osc_extent_wait()) ### extent: ffff8801c2e52498 ns: oak-OST0022-osc-ffff88041b99c000 lock: ffff880216540600/0xf077f1a82c38cbdc lrc: 3/0,0 mode: PW/PW res: [0x2a9d74:0x0:0x0].0x0 rrc: 1 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x29400000000 nid: local remote: 0xd7f0509f5e602ebb expref: -99 pid: 29353 timeout: 0 lvb_type: 1 Aug 6 07:16:29 oak-gw06 kernel: LustreError: 2091:0:(osc_cache.c:947:osc_extent_wait()) Skipped 1 previous similar message Aug 6 07:16:29 oak-gw06 kernel: LustreError: 2048:0:(osc_cache.c:947:osc_extent_wait()) extent ffff88026ba8f498@{[768 -> 868/1023], [3|0|+|rpc|wiuY|ffff88042129b560], [438272|101|+|-|ffff880143823200|256|ffff88042c4cde20]} oak-OST0010-osc-ffff88041b99c000: wait ext to 0 timedout, recovery in progress? Aug 6 07:16:29 oak-gw06 kernel: LustreError: 1983:0:(osc_cache.c:947:osc_extent_wait()) extent ffff88032f3f61f8@{[0 -> 0/255], [3|0|+|rpc|wiuY|ffff880161465690], [28672|1|+|-|ffff8803e5e44600|256|ffff88042c4cde20]} oak-OST001c-osc-ffff88041b99c000: wait ext to 0 timedout, recovery in progress? Aug 6 07:16:29 oak-gw06 kernel: LustreError: 2059:0:(osc_cache.c:947:osc_extent_wait()) extent ffff880178098bd0@{[0 -> 4/255], [3|0|+|rpc|wiuY|ffff88042129bb50], [45056|5|+|-|ffff88004328c200|256|ffff88042c4cde20]} oak-OST001a-osc-ffff88041b99c000: wait ext to 0 timedout, recovery in progress? Aug 6 07:16:29 oak-gw06 kernel: LustreError: 2063:0:(osc_cache.c:947:osc_extent_wait()) extent ffff880014497498@{[0 -> 5/255], [3|0|+|rpc|wiuY|ffff8802d9fec5f0], [49152|6|+|-|ffff88010b1eb800|256|ffff88042c4cde20]} oak-OST0000-osc-ffff88041b99c000: wait ext to 0 timedout, recovery in progress? Aug 6 07:21:33 oak-gw06 kernel: Lustre: oak-OST001a-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 6 07:21:33 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Aug 6 07:26:30 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502028989/real 1502028989] req@ffff88010d984300 x1566268698982624/t0(0) o10->oak-OST0026-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 560/432 e 1 to 1 dl 1502029590 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Aug 6 07:26:30 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Aug 6 07:26:30 oak-gw06 kernel: Lustre: oak-OST0026-osc-ffff88041b99c000: Connection to oak-OST0026 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 07:26:30 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Aug 6 07:31:34 oak-gw06 kernel: Lustre: oak-OST001a-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 6 07:31:34 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Aug 6 07:36:31 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502029590/real 1502029590] req@ffff88010d984300 x1566268698982624/t0(0) o10->oak-OST0026-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 560/432 e 1 to 1 dl 1502030191 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Aug 6 07:36:31 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Aug 6 07:36:31 oak-gw06 kernel: Lustre: oak-OST0026-osc-ffff88041b99c000: Connection to oak-OST0026 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 07:36:31 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Aug 6 07:41:35 oak-gw06 kernel: Lustre: oak-OST001a-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 6 07:41:35 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Aug 6 07:46:32 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502030191/real 1502030191] req@ffff88010d984300 x1566268698982624/t0(0) o10->oak-OST0026-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 560/432 e 1 to 1 dl 1502030792 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Aug 6 07:46:32 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Aug 6 07:46:32 oak-gw06 kernel: Lustre: oak-OST0026-osc-ffff88041b99c000: Connection to oak-OST0026 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 07:46:32 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Aug 6 07:51:36 oak-gw06 kernel: Lustre: oak-OST001a-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 6 07:51:36 oak-gw06 kernel: Lustre: Skipped 11 previous similar messages Aug 6 07:56:33 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502030792/real 1502030792] req@ffff88010d984300 x1566268698982624/t0(0) o10->oak-OST0026-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 560/432 e 1 to 1 dl 1502031393 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Aug 6 07:56:33 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Aug 6 07:56:33 oak-gw06 kernel: Lustre: oak-OST0026-osc-ffff88041b99c000: Connection to oak-OST0026 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 6 07:56:33 oak-gw06 kernel: Lustre: Skipped 11 previous similar messages Aug 6 08:01:37 oak-gw06 kernel: Lustre: oak-OST001a-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 6 08:01:37 oak-gw06 kernel: Lustre: Skipped 10 previous similar messages Aug 6 08:06:34 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502031393/real 1502031393] req@ffff88010d984300 x1566268698982624/t0(0) o10->oak-OST0026-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 560/432 e 1 to 1 dl 1502031994 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Aug 6 08:06:34 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 101 previous similar messages Aug 6 08:17:02 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1502032622/real 1502032622] req@ffff880029001200 x1566268699204384/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1502032678 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 6 08:17:02 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 272 previous similar messages Aug 6 08:25:37 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 6 08:25:37 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 6 08:25:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502032837, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880138d5a200/0xf077f1a82c38d281 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb7f8b40a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 6 08:25:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 6 08:25:37 oak-gw06 kernel: LustreError: 29652:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88042e4d0000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 6 08:25:37 oak-gw06 kernel: LustreError: 29652:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042e4d0000) refcount = 2 Aug 6 08:25:37 oak-gw06 kernel: LustreError: 29652:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 6 08:25:37 oak-gw06 kernel: LustreError: 29652:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880138d5a200/0xf077f1a82c38d281 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb7f8b40a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 6 08:25:37 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 6 08:25:37 oak-gw06 kernel: Lustre: Skipped 5 previous similar messages Aug 6 08:28:02 oak-gw06 kernel: Lustre: oak-OST0002-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 6 08:28:07 oak-gw06 kernel: LustreError: 11-0: oak-OST0004-osc-ffff88041b99c000: operation ldlm_enqueue to node 10.0.2.101@o2ib5 failed: rc = -107 Aug 6 08:28:07 oak-gw06 kernel: LustreError: Skipped 19 previous similar messages Aug 6 08:28:08 oak-gw06 kernel: LustreError: 1761:0:(import.c:671:ptlrpc_connect_import()) already connecting Aug 6 08:28:09 oak-gw06 kernel: LustreError: 11-0: oak-OST0010-osc-ffff88041b99c000: operation ldlm_enqueue to node 10.0.2.101@o2ib5 failed: rc = -107 Aug 6 08:28:09 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 6 08:28:09 oak-gw06 kernel: LustreError: 1761:0:(import.c:671:ptlrpc_connect_import()) already connecting Aug 6 08:28:09 oak-gw06 kernel: LustreError: 1761:0:(import.c:671:ptlrpc_connect_import()) Skipped 2 previous similar messages Aug 6 08:28:10 oak-gw06 kernel: LustreError: 11-0: oak-OST0000-osc-ffff88041b99c000: operation ost_write to node 10.0.2.101@o2ib5 failed: rc = -107 Aug 6 08:28:10 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Aug 6 08:28:14 oak-gw06 kernel: LustreError: 11-0: oak-OST001a-osc-ffff88041b99c000: operation ldlm_enqueue to node 10.0.2.101@o2ib5 failed: rc = -107 Aug 6 08:28:14 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 6 08:28:32 oak-gw06 kernel: LustreError: 167-0: oak-OST0000-osc-ffff88041b99c000: This client was evicted by oak-OST0000; in progress operations using this service will fail. Aug 6 08:28:57 oak-gw06 kernel: LustreError: 167-0: oak-OST0004-osc-ffff88041b99c000: This client was evicted by oak-OST0004; in progress operations using this service will fail. Aug 6 08:28:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 6 08:31:35 oak-gw06 kernel: Lustre: DEBUG MARKER: Sun Aug 6 08:31:35 2017 Aug 7 11:45:12 oak-gw06 kernel: Lustre: 31927:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502131505/real 1502131505] req@ffff88010cda8c00 x1566268941293328/t0(0) o101->oak-OST0006-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 328/400 e 0 to 1 dl 1502131512 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 7 11:45:12 oak-gw06 kernel: Lustre: 31927:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 83 previous similar messages Aug 7 11:45:12 oak-gw06 kernel: Lustre: oak-OST0006-osc-ffff88041b99c000: Connection to oak-OST0006 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 7 11:45:12 oak-gw06 kernel: Lustre: Skipped 31 previous similar messages Aug 7 11:46:13 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3126:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 10 seconds Aug 7 11:46:13 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3189:kiblnd_check_conns()) Timed out RDMA with 10.0.2.101@o2ib5 (60): c: 0, oc: 0, rc: 8 Aug 7 11:48:39 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1502131719/real 1502131719] req@ffff8802e6646100 x1566268941302464/t0(0) o8->oak-OST0008-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1502131730 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 7 11:48:39 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 41 previous similar messages Aug 7 11:51:59 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1502131919/real 1502131919] req@ffff8801a507ed00 x1566268941308128/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1502131950 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 7 11:51:59 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 83 previous similar messages Aug 7 11:56:59 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1502132219/real 1502132219] req@ffff88019f5aea00 x1566268941316688/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1502132274 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 7 11:56:59 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 125 previous similar messages Aug 7 12:00:06 oak-gw06 kernel: Lustre: oak-MDT0000-mdc-ffff88041b99c000: Connection to oak-MDT0000 (at 10.0.2.52@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 7 12:00:06 oak-gw06 kernel: Lustre: Skipped 20 previous similar messages Aug 7 12:00:06 oak-gw06 kernel: Lustre: oak-MDT0000-mdc-ffff88041b99c000: Connection restored to 10.0.2.52@o2ib5 (at 10.0.2.52@o2ib5) Aug 7 12:00:06 oak-gw06 kernel: Lustre: Skipped 20 previous similar messages Aug 7 12:08:17 oak-gw06 kernel: Lustre: oak-OST000e-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 7 12:10:09 oak-gw06 kernel: Lustre: oak-OST0002-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 7 12:10:09 oak-gw06 kernel: Lustre: Skipped 19 previous similar messages Aug 7 12:11:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 12:11:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502132760, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803db7a3c00/0xf077f1a82c64e98a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb7f9d5aa expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:11:00 oak-gw06 kernel: LustreError: 32275:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88028ff3d240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 12:11:00 oak-gw06 kernel: LustreError: 32275:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88028ff3d240) refcount = 2 Aug 7 12:11:00 oak-gw06 kernel: LustreError: 32275:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 12:11:00 oak-gw06 kernel: LustreError: 32275:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803db7a3c00/0xf077f1a82c64e98a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb7f9d5aa expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:16:10 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 12:16:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502133070, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803db7a3c00/0xf077f1a82c64f7e5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb7fa59c3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:16:10 oak-gw06 kernel: LustreError: 32288:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880051f19480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 12:16:10 oak-gw06 kernel: LustreError: 32288:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880051f19480) refcount = 2 Aug 7 12:16:10 oak-gw06 kernel: LustreError: 32288:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 12:16:10 oak-gw06 kernel: LustreError: 32288:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803db7a3c00/0xf077f1a82c64f7e5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb7fa59c3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:16:10 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 12:16:10 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 12:21:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 12:21:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502133376, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801cd677e00/0xf077f1a82c6505e5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb7facc4e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:21:16 oak-gw06 kernel: LustreError: 32310:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803097bba80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 12:21:16 oak-gw06 kernel: LustreError: 32310:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803097bba80) refcount = 2 Aug 7 12:21:16 oak-gw06 kernel: LustreError: 32310:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 12:21:16 oak-gw06 kernel: LustreError: 32310:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801cd677e00/0xf077f1a82c6505e5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb7facc4e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:21:16 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 12:26:24 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 12:26:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502133684, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801cd676e00/0xf077f1a82c6525c7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb7fb49ad expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:26:24 oak-gw06 kernel: LustreError: 32315:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88014c884c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 12:26:24 oak-gw06 kernel: LustreError: 32315:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88014c884c00) refcount = 2 Aug 7 12:26:24 oak-gw06 kernel: LustreError: 32315:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 12:26:24 oak-gw06 kernel: LustreError: 32315:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801cd676e00/0xf077f1a82c6525c7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb7fb49ad expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:26:24 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 12:31:33 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 12:31:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502133993, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880277baba00/0xf077f1a82c6543d4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb7fbc537 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:31:33 oak-gw06 kernel: LustreError: 32327:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801c657e240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 12:31:33 oak-gw06 kernel: LustreError: 32327:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c657e240) refcount = 2 Aug 7 12:31:33 oak-gw06 kernel: LustreError: 32327:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 12:31:33 oak-gw06 kernel: LustreError: 32327:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880277baba00/0xf077f1a82c6543d4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb7fbc537 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:36:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 12:36:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502134302, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880153a27000/0xf077f1a82c656aee lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb7fc449c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:36:42 oak-gw06 kernel: LustreError: 32331:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880065626d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 12:36:42 oak-gw06 kernel: LustreError: 32331:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880065626d80) refcount = 2 Aug 7 12:36:42 oak-gw06 kernel: LustreError: 32331:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 12:36:42 oak-gw06 kernel: LustreError: 32331:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880153a27000/0xf077f1a82c656aee lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb7fc449c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:36:42 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 12:36:42 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 12:41:47 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 12:41:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502134607, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880072f24000/0xf077f1a82c657edd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb7fcb36f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:41:47 oak-gw06 kernel: LustreError: 32343:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880065626300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 12:41:47 oak-gw06 kernel: LustreError: 32343:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880065626300) refcount = 2 Aug 7 12:41:47 oak-gw06 kernel: LustreError: 32343:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 12:41:47 oak-gw06 kernel: LustreError: 32343:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880072f24000/0xf077f1a82c657edd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb7fcb36f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:46:54 oak-gw06 kernel: LustreError: 32347:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880065626900) refcount = 2 Aug 7 12:46:54 oak-gw06 kernel: LustreError: 32347:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 12:46:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 12:46:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 12:52:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 12:52:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 12:52:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502135220, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880072f25e00/0xf077f1a82c657fb6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb7fd944f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:52:00 oak-gw06 kernel: LustreError: 32359:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880065626300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 12:52:00 oak-gw06 kernel: LustreError: 32359:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 12:52:00 oak-gw06 kernel: LustreError: 32359:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880065626300) refcount = 2 Aug 7 12:52:00 oak-gw06 kernel: LustreError: 32359:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 12:52:00 oak-gw06 kernel: LustreError: 32359:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880072f25e00/0xf077f1a82c657fb6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb7fd944f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 12:52:00 oak-gw06 kernel: LustreError: 32359:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 12:52:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 12:57:06 oak-gw06 kernel: LustreError: 32363:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880065626300) refcount = 2 Aug 7 12:57:06 oak-gw06 kernel: LustreError: 32363:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 12:57:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 12:57:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 13:02:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 13:02:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 13:02:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502135833, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880072f27000/0xf077f1a82c65825d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb7fe75de expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 13:02:13 oak-gw06 kernel: LustreError: 32406:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800656269c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 13:02:13 oak-gw06 kernel: LustreError: 32406:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 13:02:13 oak-gw06 kernel: LustreError: 32406:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800656269c0) refcount = 2 Aug 7 13:02:13 oak-gw06 kernel: LustreError: 32406:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 13:02:13 oak-gw06 kernel: LustreError: 32406:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880072f27000/0xf077f1a82c65825d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb7fe75de expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 13:02:13 oak-gw06 kernel: LustreError: 32406:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 13:02:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 13:07:18 oak-gw06 kernel: LustreError: 32410:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880065626240) refcount = 2 Aug 7 13:07:18 oak-gw06 kernel: LustreError: 32410:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 13:07:18 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 13:07:18 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 13:12:24 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 13:12:24 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 13:12:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502136444, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880072f25c00/0xf077f1a82c658295 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb7ff4d9c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 13:12:24 oak-gw06 kernel: LustreError: 32425:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880065626b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 13:12:24 oak-gw06 kernel: LustreError: 32425:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 13:12:24 oak-gw06 kernel: LustreError: 32425:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880065626b40) refcount = 2 Aug 7 13:12:24 oak-gw06 kernel: LustreError: 32425:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 13:12:24 oak-gw06 kernel: LustreError: 32425:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880072f25c00/0xf077f1a82c658295 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb7ff4d9c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 13:12:24 oak-gw06 kernel: LustreError: 32425:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 13:12:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 13:17:31 oak-gw06 kernel: LustreError: 32429:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c657e240) refcount = 2 Aug 7 13:17:31 oak-gw06 kernel: LustreError: 32429:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 13:17:31 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 13:17:31 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 13:22:37 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 13:22:37 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 13:22:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502137057, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880209e4e400/0xf077f1a82c65b627 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb800376d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 13:22:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 13:22:37 oak-gw06 kernel: LustreError: 32441:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88017a091780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 13:22:37 oak-gw06 kernel: LustreError: 32441:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 13:22:37 oak-gw06 kernel: LustreError: 32441:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88017a091780) refcount = 2 Aug 7 13:22:37 oak-gw06 kernel: LustreError: 32441:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 13:22:37 oak-gw06 kernel: LustreError: 32441:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880209e4e400/0xf077f1a82c65b627 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb800376d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 13:22:37 oak-gw06 kernel: LustreError: 32441:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 13:27:46 oak-gw06 kernel: LustreError: 32449:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802603a4900) refcount = 2 Aug 7 13:27:46 oak-gw06 kernel: LustreError: 32449:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 13:27:46 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 13:27:46 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 13:32:52 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 13:32:52 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 13:32:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502137672, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801cd675000/0xf077f1a82c65ea45 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb80125dd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 13:32:52 oak-gw06 kernel: LustreError: 32462:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802757fc480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 13:32:52 oak-gw06 kernel: LustreError: 32462:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 13:32:52 oak-gw06 kernel: LustreError: 32462:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802757fc480) refcount = 2 Aug 7 13:32:52 oak-gw06 kernel: LustreError: 32462:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 13:32:52 oak-gw06 kernel: LustreError: 32462:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801cd675000/0xf077f1a82c65ea45 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb80125dd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 13:32:52 oak-gw06 kernel: LustreError: 32462:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 13:32:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 13:38:01 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 13:38:01 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 13:43:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 13:43:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 13:43:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502138291, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802ac373200/0xf077f1a82c6638f7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb802298c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 13:43:11 oak-gw06 kernel: LustreError: 32481:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880421a933c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 13:43:11 oak-gw06 kernel: LustreError: 32481:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880421a933c0) refcount = 2 Aug 7 13:43:11 oak-gw06 kernel: LustreError: 32481:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 13:43:11 oak-gw06 kernel: LustreError: 32481:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802ac373200/0xf077f1a82c6638f7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb802298c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 13:43:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 13:48:21 oak-gw06 kernel: LustreError: 32485:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88028e3fdf00) refcount = 2 Aug 7 13:48:21 oak-gw06 kernel: LustreError: 32485:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 13:48:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 13:48:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 13:53:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 13:53:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 13:53:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502138908, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88016c778400/0xf077f1a82c668d60 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb803277d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 13:53:28 oak-gw06 kernel: LustreError: 32499:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88022d4f1240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 13:53:28 oak-gw06 kernel: LustreError: 32499:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 13:53:28 oak-gw06 kernel: LustreError: 32499:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022d4f1240) refcount = 2 Aug 7 13:53:28 oak-gw06 kernel: LustreError: 32499:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 13:53:28 oak-gw06 kernel: LustreError: 32499:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88016c778400/0xf077f1a82c668d60 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb803277d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 13:53:28 oak-gw06 kernel: LustreError: 32499:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 13:53:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 13:58:35 oak-gw06 kernel: LustreError: 32502:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880412ef5a80) refcount = 2 Aug 7 13:58:35 oak-gw06 kernel: LustreError: 32502:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 13:58:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 13:58:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 14:03:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 14:03:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 14:03:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502139522, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880197eaa200/0xf077f1a82c66d93a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb804141f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 14:03:42 oak-gw06 kernel: LustreError: 32550:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800b8ec7f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 14:03:42 oak-gw06 kernel: LustreError: 32550:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 14:03:42 oak-gw06 kernel: LustreError: 32550:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800b8ec7f00) refcount = 2 Aug 7 14:03:42 oak-gw06 kernel: LustreError: 32550:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 14:03:42 oak-gw06 kernel: LustreError: 32550:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880197eaa200/0xf077f1a82c66d93a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb804141f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 14:03:42 oak-gw06 kernel: LustreError: 32550:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 14:03:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 14:08:48 oak-gw06 kernel: LustreError: 32554:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880374aea780) refcount = 2 Aug 7 14:08:48 oak-gw06 kernel: LustreError: 32554:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 14:08:48 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 14:08:48 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 14:13:56 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 14:13:56 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 14:13:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502140136, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800b9cf7600/0xf077f1a82c6729e4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb80500d6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 14:13:56 oak-gw06 kernel: LustreError: 32569:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88005a0a63c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 14:13:56 oak-gw06 kernel: LustreError: 32569:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 14:13:56 oak-gw06 kernel: LustreError: 32569:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88005a0a63c0) refcount = 2 Aug 7 14:13:56 oak-gw06 kernel: LustreError: 32569:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 14:13:56 oak-gw06 kernel: LustreError: 32569:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800b9cf7600/0xf077f1a82c6729e4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb80500d6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 14:13:56 oak-gw06 kernel: LustreError: 32569:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 14:13:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 14:19:05 oak-gw06 kernel: LustreError: 32573:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88019584a540) refcount = 2 Aug 7 14:19:05 oak-gw06 kernel: LustreError: 32573:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 14:19:05 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 14:19:05 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 14:24:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 14:24:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 14:24:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502140754, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880072f27a00/0xf077f1a82c67691c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb805fd69 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 14:24:14 oak-gw06 kernel: LustreError: 32600:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803eb78f840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 14:24:14 oak-gw06 kernel: LustreError: 32600:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 14:24:14 oak-gw06 kernel: LustreError: 32600:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803eb78f840) refcount = 2 Aug 7 14:24:14 oak-gw06 kernel: LustreError: 32600:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 14:24:14 oak-gw06 kernel: LustreError: 32600:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880072f27a00/0xf077f1a82c67691c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb805fd69 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 14:24:14 oak-gw06 kernel: LustreError: 32600:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 14:24:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 14:29:22 oak-gw06 kernel: LustreError: 32608:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88023a2fef00) refcount = 2 Aug 7 14:29:22 oak-gw06 kernel: LustreError: 32608:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 14:29:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 14:29:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 14:34:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 14:34:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 14:34:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502141371, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880190b53e00/0xf077f1a82c6801fd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb806f970 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 14:34:31 oak-gw06 kernel: LustreError: 32621:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88035ff4f600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 14:34:31 oak-gw06 kernel: LustreError: 32621:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 14:34:31 oak-gw06 kernel: LustreError: 32621:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88035ff4f600) refcount = 2 Aug 7 14:34:31 oak-gw06 kernel: LustreError: 32621:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 14:34:31 oak-gw06 kernel: LustreError: 32621:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880190b53e00/0xf077f1a82c6801fd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb806f970 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 14:34:31 oak-gw06 kernel: LustreError: 32621:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 14:34:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 14:39:39 oak-gw06 kernel: LustreError: 32624:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880046a2d6c0) refcount = 2 Aug 7 14:39:39 oak-gw06 kernel: LustreError: 32624:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 14:39:39 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 14:39:39 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 14:44:46 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 14:44:46 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 14:44:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502141986, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802ac371800/0xf077f1a82c683278 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb807e7e7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 14:44:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 14:44:46 oak-gw06 kernel: LustreError: 32637:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880046a2d780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 14:44:46 oak-gw06 kernel: LustreError: 32637:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 14:44:46 oak-gw06 kernel: LustreError: 32637:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880046a2d780) refcount = 2 Aug 7 14:44:46 oak-gw06 kernel: LustreError: 32637:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 14:44:46 oak-gw06 kernel: LustreError: 32637:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802ac371800/0xf077f1a82c683278 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb807e7e7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 14:44:46 oak-gw06 kernel: LustreError: 32637:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 14:49:55 oak-gw06 kernel: LustreError: 32644:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88000eabe180) refcount = 2 Aug 7 14:49:55 oak-gw06 kernel: LustreError: 32644:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 14:49:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 14:49:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 14:55:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 14:55:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 14:55:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502142602, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88029ee4ac00/0xf077f1a82c68588f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb808dc23 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 14:55:02 oak-gw06 kernel: LustreError: 32669:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801e8a36780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 14:55:02 oak-gw06 kernel: LustreError: 32669:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 14:55:02 oak-gw06 kernel: LustreError: 32669:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801e8a36780) refcount = 2 Aug 7 14:55:02 oak-gw06 kernel: LustreError: 32669:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 14:55:02 oak-gw06 kernel: LustreError: 32669:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88029ee4ac00/0xf077f1a82c68588f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb808dc23 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 14:55:02 oak-gw06 kernel: LustreError: 32669:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 14:55:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 15:00:10 oak-gw06 kernel: LustreError: 32690:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88035e977840) refcount = 2 Aug 7 15:00:10 oak-gw06 kernel: LustreError: 32690:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 15:00:10 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 15:00:10 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 15:05:20 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 15:05:20 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 15:05:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502143220, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880356a1b600/0xf077f1a82c695290 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb809d789 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 15:05:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 15:05:20 oak-gw06 kernel: LustreError: 32729:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ccb1b900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 15:05:20 oak-gw06 kernel: LustreError: 32729:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 15:05:20 oak-gw06 kernel: LustreError: 32729:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ccb1b900) refcount = 2 Aug 7 15:05:20 oak-gw06 kernel: LustreError: 32729:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 15:05:20 oak-gw06 kernel: LustreError: 32729:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880356a1b600/0xf077f1a82c695290 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb809d789 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 15:05:20 oak-gw06 kernel: LustreError: 32729:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 15:10:27 oak-gw06 kernel: LustreError: 32750:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88024b7f7b40) refcount = 2 Aug 7 15:10:27 oak-gw06 kernel: LustreError: 32750:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 15:10:27 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 15:10:27 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 15:15:37 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 15:15:37 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 15:15:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502143837, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88029823ba00/0xf077f1a82c6a784a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb80acf84 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 15:15:37 oak-gw06 kernel: LustreError: 32762:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802ee668300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 15:15:37 oak-gw06 kernel: LustreError: 32762:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 15:15:37 oak-gw06 kernel: LustreError: 32762:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802ee668300) refcount = 2 Aug 7 15:15:37 oak-gw06 kernel: LustreError: 32762:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 15:15:37 oak-gw06 kernel: LustreError: 32762:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88029823ba00/0xf077f1a82c6a784a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb80acf84 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 15:15:37 oak-gw06 kernel: LustreError: 32762:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 15:15:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 15:20:45 oak-gw06 kernel: LustreError: 317:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880313b76000) refcount = 2 Aug 7 15:20:45 oak-gw06 kernel: LustreError: 317:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 15:20:45 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 15:20:45 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 15:25:51 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 15:25:51 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 15:25:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502144451, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88016a6c6200/0xf077f1a82c6b966a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb80bbfa6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 15:25:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 15:25:51 oak-gw06 kernel: LustreError: 325:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801b92ac0c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 15:25:51 oak-gw06 kernel: LustreError: 325:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 15:25:51 oak-gw06 kernel: LustreError: 325:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801b92ac0c0) refcount = 2 Aug 7 15:25:51 oak-gw06 kernel: LustreError: 325:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 15:25:51 oak-gw06 kernel: LustreError: 325:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88016a6c6200/0xf077f1a82c6b966a lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb80bbfa6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 15:25:51 oak-gw06 kernel: LustreError: 325:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 15:31:00 oak-gw06 kernel: LustreError: 345:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88021070fd80) refcount = 2 Aug 7 15:31:00 oak-gw06 kernel: LustreError: 345:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 15:31:00 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 15:31:00 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 15:36:09 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 15:36:09 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 15:36:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502145069, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880209977a00/0xf077f1a82c6bc104 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb80cb976 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 15:36:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 15:36:09 oak-gw06 kernel: LustreError: 352:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802f7f796c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 15:36:09 oak-gw06 kernel: LustreError: 352:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 15:36:09 oak-gw06 kernel: LustreError: 352:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802f7f796c0) refcount = 2 Aug 7 15:36:09 oak-gw06 kernel: LustreError: 352:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 15:36:09 oak-gw06 kernel: LustreError: 352:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880209977a00/0xf077f1a82c6bc104 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb80cb976 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 15:36:09 oak-gw06 kernel: LustreError: 352:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 15:41:14 oak-gw06 kernel: LustreError: 434:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ca3b6a80) refcount = 2 Aug 7 15:41:14 oak-gw06 kernel: LustreError: 434:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 15:41:14 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 15:41:14 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 15:46:21 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 15:46:21 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 15:46:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502145681, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801a9269e00/0xf077f1a82c718d3c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb80da1c6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 15:46:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 15:46:21 oak-gw06 kernel: LustreError: 439:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802512e7cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 15:46:21 oak-gw06 kernel: LustreError: 439:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 15:46:21 oak-gw06 kernel: LustreError: 439:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802512e7cc0) refcount = 2 Aug 7 15:46:21 oak-gw06 kernel: LustreError: 439:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 15:46:21 oak-gw06 kernel: LustreError: 439:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801a9269e00/0xf077f1a82c718d3c lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb80da1c6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 15:46:21 oak-gw06 kernel: LustreError: 439:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 15:51:28 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 15:51:28 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 15:56:37 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 15:56:37 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 15:56:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502146297, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880092e47200/0xf077f1a82c721dfe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb80e941f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 15:56:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 15:56:37 oak-gw06 kernel: LustreError: 468:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ba2aba80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 15:56:37 oak-gw06 kernel: LustreError: 468:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ba2aba80) refcount = 2 Aug 7 15:56:37 oak-gw06 kernel: LustreError: 468:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 15:56:37 oak-gw06 kernel: LustreError: 468:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880092e47200/0xf077f1a82c721dfe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb80e941f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 16:01:47 oak-gw06 kernel: LustreError: 515:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801af316780) refcount = 2 Aug 7 16:01:47 oak-gw06 kernel: LustreError: 515:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 16:01:47 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 16:01:47 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 16:06:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 16:06:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 16:06:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502146917, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880080e2ca00/0xf077f1a82c72d8ce lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb80f933d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 16:06:57 oak-gw06 kernel: LustreError: 519:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880421a93480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 16:06:57 oak-gw06 kernel: LustreError: 519:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 16:06:57 oak-gw06 kernel: LustreError: 519:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880421a93480) refcount = 2 Aug 7 16:06:57 oak-gw06 kernel: LustreError: 519:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 16:06:57 oak-gw06 kernel: LustreError: 519:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880080e2ca00/0xf077f1a82c72d8ce lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb80f933d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 16:06:57 oak-gw06 kernel: LustreError: 519:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 16:06:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 16:12:07 oak-gw06 kernel: LustreError: 535:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88003bbe3900) refcount = 2 Aug 7 16:12:07 oak-gw06 kernel: LustreError: 535:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 16:12:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 16:12:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 16:17:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 16:17:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 16:17:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502147533, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802fc845800/0xf077f1a82c731f14 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8108596 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 16:17:13 oak-gw06 kernel: LustreError: 538:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802c8b8e240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 16:17:13 oak-gw06 kernel: LustreError: 538:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 16:17:13 oak-gw06 kernel: LustreError: 538:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802c8b8e240) refcount = 2 Aug 7 16:17:13 oak-gw06 kernel: LustreError: 538:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 16:17:13 oak-gw06 kernel: LustreError: 538:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802fc845800/0xf077f1a82c731f14 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8108596 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 16:17:13 oak-gw06 kernel: LustreError: 538:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 16:17:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 16:22:20 oak-gw06 kernel: LustreError: 550:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88011dc18780) refcount = 2 Aug 7 16:22:20 oak-gw06 kernel: LustreError: 550:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 16:22:20 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 16:22:20 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 16:27:26 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 16:27:26 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 16:27:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502148146, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880072c57000/0xf077f1a82c735577 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8117078 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 16:27:26 oak-gw06 kernel: LustreError: 557:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88018e2d3b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 16:27:26 oak-gw06 kernel: LustreError: 557:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 16:27:26 oak-gw06 kernel: LustreError: 557:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88018e2d3b40) refcount = 2 Aug 7 16:27:26 oak-gw06 kernel: LustreError: 557:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 16:27:26 oak-gw06 kernel: LustreError: 557:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880072c57000/0xf077f1a82c735577 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8117078 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 16:27:26 oak-gw06 kernel: LustreError: 557:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 16:27:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 16:32:31 oak-gw06 kernel: LustreError: 569:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88020e0d6000) refcount = 2 Aug 7 16:32:31 oak-gw06 kernel: LustreError: 569:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 16:32:31 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 16:32:31 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 16:37:38 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 16:37:38 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 16:37:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502148758, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800184f8200/0xf077f1a82c73a94d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8125938 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 16:37:38 oak-gw06 kernel: LustreError: 576:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800055ec000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 16:37:38 oak-gw06 kernel: LustreError: 576:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 16:37:38 oak-gw06 kernel: LustreError: 576:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800055ec000) refcount = 2 Aug 7 16:37:38 oak-gw06 kernel: LustreError: 576:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 16:37:38 oak-gw06 kernel: LustreError: 576:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800184f8200/0xf077f1a82c73a94d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8125938 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 16:37:38 oak-gw06 kernel: LustreError: 576:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 16:37:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 16:42:45 oak-gw06 kernel: LustreError: 599:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880205bad3c0) refcount = 2 Aug 7 16:42:45 oak-gw06 kernel: LustreError: 599:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 16:42:45 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 16:42:45 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 16:47:51 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 16:47:51 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 16:47:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502149371, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801a55f3000/0xf077f1a82c742149 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8134007 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 16:47:51 oak-gw06 kernel: LustreError: 610:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801e6ed1cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 16:47:51 oak-gw06 kernel: LustreError: 610:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 16:47:51 oak-gw06 kernel: LustreError: 610:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801e6ed1cc0) refcount = 2 Aug 7 16:47:51 oak-gw06 kernel: LustreError: 610:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 16:47:51 oak-gw06 kernel: LustreError: 610:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801a55f3000/0xf077f1a82c742149 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8134007 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 16:47:51 oak-gw06 kernel: LustreError: 610:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 16:47:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 16:53:00 oak-gw06 kernel: LustreError: 633:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801e6ed1900) refcount = 2 Aug 7 16:53:00 oak-gw06 kernel: LustreError: 633:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 16:53:00 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 16:53:00 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 16:58:05 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 16:58:05 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 16:58:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502149985, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88016ffdca00/0xf077f1a82c74a16b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8142eb6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 16:58:05 oak-gw06 kernel: LustreError: 636:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880233f4d6c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 16:58:05 oak-gw06 kernel: LustreError: 636:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 16:58:05 oak-gw06 kernel: LustreError: 636:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880233f4d6c0) refcount = 2 Aug 7 16:58:05 oak-gw06 kernel: LustreError: 636:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 16:58:05 oak-gw06 kernel: LustreError: 636:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88016ffdca00/0xf077f1a82c74a16b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8142eb6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 16:58:05 oak-gw06 kernel: LustreError: 636:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 16:58:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 17:03:15 oak-gw06 kernel: LustreError: 703:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880233f4df00) refcount = 2 Aug 7 17:03:15 oak-gw06 kernel: LustreError: 703:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 17:03:15 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 17:03:15 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 17:08:24 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 17:08:24 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 17:08:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502150604, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802a3664600/0xf077f1a82c752c6f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8152c99 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 17:08:24 oak-gw06 kernel: LustreError: 712:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88035fff1540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 17:08:24 oak-gw06 kernel: LustreError: 712:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 17:08:24 oak-gw06 kernel: LustreError: 712:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88035fff1540) refcount = 2 Aug 7 17:08:24 oak-gw06 kernel: LustreError: 712:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 17:08:24 oak-gw06 kernel: LustreError: 712:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802a3664600/0xf077f1a82c752c6f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8152c99 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 17:08:24 oak-gw06 kernel: LustreError: 712:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 17:08:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 17:13:33 oak-gw06 kernel: LustreError: 722:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022a219c00) refcount = 2 Aug 7 17:13:33 oak-gw06 kernel: LustreError: 722:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 17:13:33 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 17:13:33 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 17:18:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 17:18:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 17:18:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502151221, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880134302800/0xf077f1a82c75d189 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb81621f4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 17:18:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 17:18:41 oak-gw06 kernel: LustreError: 730:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880231e2d9c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 17:18:41 oak-gw06 kernel: LustreError: 730:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 17:18:41 oak-gw06 kernel: LustreError: 730:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880231e2d9c0) refcount = 2 Aug 7 17:18:41 oak-gw06 kernel: LustreError: 730:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 17:18:41 oak-gw06 kernel: LustreError: 730:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880134302800/0xf077f1a82c75d189 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb81621f4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 17:18:41 oak-gw06 kernel: LustreError: 730:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 17:23:49 oak-gw06 kernel: LustreError: 742:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ae9169c0) refcount = 2 Aug 7 17:23:49 oak-gw06 kernel: LustreError: 742:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 17:23:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 17:23:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 17:28:55 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 17:28:55 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 17:28:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502151835, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88004196d000/0xf077f1a82c76065d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8170e49 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 17:28:55 oak-gw06 kernel: LustreError: 746:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88020287aa80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 17:28:55 oak-gw06 kernel: LustreError: 746:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 17:28:55 oak-gw06 kernel: LustreError: 746:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88020287aa80) refcount = 2 Aug 7 17:28:55 oak-gw06 kernel: LustreError: 746:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 17:28:55 oak-gw06 kernel: LustreError: 746:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88004196d000/0xf077f1a82c76065d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8170e49 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 17:28:55 oak-gw06 kernel: LustreError: 746:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 17:28:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 17:34:04 oak-gw06 kernel: LustreError: 761:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801bd6dcb40) refcount = 2 Aug 7 17:34:04 oak-gw06 kernel: LustreError: 761:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 17:34:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 17:34:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 17:39:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 17:39:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 17:39:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502152454, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802412e7c00/0xf077f1a82c7657d2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8180b30 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 17:39:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 17:39:14 oak-gw06 kernel: LustreError: 768:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803af186c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 17:39:14 oak-gw06 kernel: LustreError: 768:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 17:39:14 oak-gw06 kernel: LustreError: 768:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803af186c00) refcount = 2 Aug 7 17:39:14 oak-gw06 kernel: LustreError: 768:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 17:39:14 oak-gw06 kernel: LustreError: 768:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802412e7c00/0xf077f1a82c7657d2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8180b30 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 17:39:14 oak-gw06 kernel: LustreError: 768:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 17:44:19 oak-gw06 kernel: LustreError: 780:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803daf55180) refcount = 2 Aug 7 17:44:19 oak-gw06 kernel: LustreError: 780:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 17:44:19 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 17:44:19 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 17:49:25 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 17:49:25 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 17:49:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502153065, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880171c73600/0xf077f1a82c76b303 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb818f19d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 17:49:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 17:49:25 oak-gw06 kernel: LustreError: 787:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802690eee40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 17:49:25 oak-gw06 kernel: LustreError: 787:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 17:49:25 oak-gw06 kernel: LustreError: 787:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802690eee40) refcount = 2 Aug 7 17:49:25 oak-gw06 kernel: LustreError: 787:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 17:49:25 oak-gw06 kernel: LustreError: 787:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880171c73600/0xf077f1a82c76b303 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb818f19d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 17:49:25 oak-gw06 kernel: LustreError: 787:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 17:54:35 oak-gw06 kernel: LustreError: 798:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88037aaca000) refcount = 2 Aug 7 17:54:35 oak-gw06 kernel: LustreError: 798:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 17:54:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 17:54:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 17:59:43 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 17:59:43 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 17:59:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502153683, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880190e60a00/0xf077f1a82c770631 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb819e887 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 17:59:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 17:59:43 oak-gw06 kernel: LustreError: 806:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880362bce0c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 17:59:43 oak-gw06 kernel: LustreError: 806:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 17:59:43 oak-gw06 kernel: LustreError: 806:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880362bce0c0) refcount = 2 Aug 7 17:59:43 oak-gw06 kernel: LustreError: 806:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 17:59:43 oak-gw06 kernel: LustreError: 806:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880190e60a00/0xf077f1a82c770631 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb819e887 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 17:59:43 oak-gw06 kernel: LustreError: 806:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 18:04:49 oak-gw06 kernel: LustreError: 851:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880365a07480) refcount = 2 Aug 7 18:04:49 oak-gw06 kernel: LustreError: 851:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 18:04:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 18:04:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 18:09:56 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 18:09:56 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 18:09:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502154296, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803216bf000/0xf077f1a82c776a5a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb81ad124 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 18:09:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 18:09:56 oak-gw06 kernel: LustreError: 858:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e56b70c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 18:09:56 oak-gw06 kernel: LustreError: 858:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 18:09:56 oak-gw06 kernel: LustreError: 858:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e56b70c0) refcount = 2 Aug 7 18:09:56 oak-gw06 kernel: LustreError: 858:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 18:09:56 oak-gw06 kernel: LustreError: 858:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803216bf000/0xf077f1a82c776a5a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb81ad124 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 18:09:56 oak-gw06 kernel: LustreError: 858:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 18:15:02 oak-gw06 kernel: LustreError: 871:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802b03353c0) refcount = 2 Aug 7 18:15:02 oak-gw06 kernel: LustreError: 871:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 18:15:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 18:15:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 18:20:12 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 18:20:12 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 18:20:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502154912, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802ebec6c00/0xf077f1a82c77ce13 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb81bc583 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 18:20:12 oak-gw06 kernel: LustreError: 886:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800832e3480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 18:20:12 oak-gw06 kernel: LustreError: 886:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 18:20:12 oak-gw06 kernel: LustreError: 886:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800832e3480) refcount = 2 Aug 7 18:20:12 oak-gw06 kernel: LustreError: 886:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 18:20:12 oak-gw06 kernel: LustreError: 886:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802ebec6c00/0xf077f1a82c77ce13 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb81bc583 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 18:20:12 oak-gw06 kernel: LustreError: 886:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 18:20:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 18:25:21 oak-gw06 kernel: LustreError: 889:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802ad7be000) refcount = 2 Aug 7 18:25:21 oak-gw06 kernel: LustreError: 889:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 18:25:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 18:25:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 18:30:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 18:30:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 18:30:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502155531, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88022e047800/0xf077f1a82c780f89 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb81cc191 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 18:30:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 18:30:31 oak-gw06 kernel: LustreError: 904:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88002237d9c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 18:30:31 oak-gw06 kernel: LustreError: 904:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 18:30:31 oak-gw06 kernel: LustreError: 904:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88002237d9c0) refcount = 2 Aug 7 18:30:31 oak-gw06 kernel: LustreError: 904:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 18:30:31 oak-gw06 kernel: LustreError: 904:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88022e047800/0xf077f1a82c780f89 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb81cc191 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 18:30:31 oak-gw06 kernel: LustreError: 904:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 18:35:38 oak-gw06 kernel: LustreError: 912:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff88035e977180) refcount = 2 Aug 7 18:35:38 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 18:35:38 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 18:40:46 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 18:40:46 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 18:40:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502156146, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803b2f3bc00/0xf077f1a82c78af24 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb81db071 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 18:40:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 18:40:46 oak-gw06 kernel: LustreError: 926:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88035e977d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 18:40:46 oak-gw06 kernel: LustreError: 926:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 18:40:46 oak-gw06 kernel: LustreError: 926:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88035e977d80) refcount = 2 Aug 7 18:40:46 oak-gw06 kernel: LustreError: 926:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 18:40:46 oak-gw06 kernel: LustreError: 926:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803b2f3bc00/0xf077f1a82c78af24 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb81db071 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 18:45:55 oak-gw06 kernel: LustreError: 931:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803af186d80) refcount = 2 Aug 7 18:45:55 oak-gw06 kernel: LustreError: 931:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 18:45:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 18:45:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 18:51:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 18:51:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 18:51:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502156762, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801d3e54800/0xf077f1a82c7900b5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb81ea276 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 18:51:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 18:51:02 oak-gw06 kernel: LustreError: 944:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880205bad240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 18:51:02 oak-gw06 kernel: LustreError: 944:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 18:51:02 oak-gw06 kernel: LustreError: 944:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880205bad240) refcount = 2 Aug 7 18:51:02 oak-gw06 kernel: LustreError: 944:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 18:51:02 oak-gw06 kernel: LustreError: 944:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801d3e54800/0xf077f1a82c7900b5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb81ea276 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 18:51:02 oak-gw06 kernel: LustreError: 944:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 18:56:09 oak-gw06 kernel: LustreError: 952:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88027b307540) refcount = 2 Aug 7 18:56:09 oak-gw06 kernel: LustreError: 952:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 18:56:09 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 18:56:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 19:01:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 19:01:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 19:01:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502157374, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801fe1a1c00/0xf077f1a82c794423 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb81f8f42 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 19:01:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 19:01:14 oak-gw06 kernel: LustreError: 994:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88035f26b840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 19:01:14 oak-gw06 kernel: LustreError: 994:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 19:01:14 oak-gw06 kernel: LustreError: 994:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88035f26b840) refcount = 2 Aug 7 19:01:14 oak-gw06 kernel: LustreError: 994:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 19:01:14 oak-gw06 kernel: LustreError: 994:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801fe1a1c00/0xf077f1a82c794423 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb81f8f42 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 19:01:14 oak-gw06 kernel: LustreError: 994:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 19:06:24 oak-gw06 kernel: LustreError: 999:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880345a41a80) refcount = 2 Aug 7 19:06:24 oak-gw06 kernel: LustreError: 999:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 19:06:24 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 19:06:24 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 19:11:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 19:11:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 19:11:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502157991, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880036da9800/0xf077f1a82c797db2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8208385 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 19:11:31 oak-gw06 kernel: LustreError: 1014:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88033c7423c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 19:11:31 oak-gw06 kernel: LustreError: 1014:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 19:11:31 oak-gw06 kernel: LustreError: 1014:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033c7423c0) refcount = 2 Aug 7 19:11:31 oak-gw06 kernel: LustreError: 1014:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 19:11:31 oak-gw06 kernel: LustreError: 1014:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880036da9800/0xf077f1a82c797db2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8208385 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 19:11:31 oak-gw06 kernel: LustreError: 1014:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 19:11:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 19:16:40 oak-gw06 kernel: LustreError: 1022:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802252a23c0) refcount = 2 Aug 7 19:16:40 oak-gw06 kernel: LustreError: 1022:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 19:16:40 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 19:16:40 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 19:21:46 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 19:21:46 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 19:21:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502158606, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88032f302600/0xf077f1a82c7a0f00 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8217559 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 19:21:46 oak-gw06 kernel: LustreError: 1042:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802a267e600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 19:21:46 oak-gw06 kernel: LustreError: 1042:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 19:21:46 oak-gw06 kernel: LustreError: 1042:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a267e600) refcount = 2 Aug 7 19:21:46 oak-gw06 kernel: LustreError: 1042:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 19:21:46 oak-gw06 kernel: LustreError: 1042:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88032f302600/0xf077f1a82c7a0f00 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8217559 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 19:21:46 oak-gw06 kernel: LustreError: 1042:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 19:21:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 19:26:54 oak-gw06 kernel: LustreError: 1049:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801bd6dcf00) refcount = 2 Aug 7 19:26:54 oak-gw06 kernel: LustreError: 1049:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 19:26:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 19:26:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 19:32:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 19:32:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 19:32:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502159220, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880134303c00/0xf077f1a82c7b13e3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb82263bb expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 19:32:00 oak-gw06 kernel: LustreError: 1059:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802de499b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 19:32:00 oak-gw06 kernel: LustreError: 1059:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 19:32:00 oak-gw06 kernel: LustreError: 1059:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802de499b40) refcount = 2 Aug 7 19:32:00 oak-gw06 kernel: LustreError: 1059:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 19:32:00 oak-gw06 kernel: LustreError: 1059:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880134303c00/0xf077f1a82c7b13e3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb82263bb expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 19:32:00 oak-gw06 kernel: LustreError: 1059:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 19:32:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 19:37:10 oak-gw06 kernel: LustreError: 1068:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ca3b63c0) refcount = 2 Aug 7 19:37:10 oak-gw06 kernel: LustreError: 1068:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 19:37:10 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 19:37:10 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 19:42:17 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 19:42:17 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 19:42:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502159837, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880357146e00/0xf077f1a82c7b5eb3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb82359f6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 19:42:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 19:42:17 oak-gw06 kernel: LustreError: 1079:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a243b000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 19:42:17 oak-gw06 kernel: LustreError: 1079:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 19:42:17 oak-gw06 kernel: LustreError: 1079:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a243b000) refcount = 2 Aug 7 19:42:17 oak-gw06 kernel: LustreError: 1079:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 19:42:17 oak-gw06 kernel: LustreError: 1079:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880357146e00/0xf077f1a82c7b5eb3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb82359f6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 19:42:17 oak-gw06 kernel: LustreError: 1079:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 19:47:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 19:47:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 19:52:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 19:52:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 19:52:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502160448, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800130c0c00/0xf077f1a82c7d0775 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb82441c1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 19:52:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 19:52:28 oak-gw06 kernel: LustreError: 1114:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803096b9900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 19:52:28 oak-gw06 kernel: LustreError: 1114:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803096b9900) refcount = 2 Aug 7 19:52:28 oak-gw06 kernel: LustreError: 1114:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 19:52:28 oak-gw06 kernel: LustreError: 1114:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800130c0c00/0xf077f1a82c7d0775 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb82441c1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 19:57:36 oak-gw06 kernel: LustreError: 1118:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802fae90600) refcount = 2 Aug 7 19:57:36 oak-gw06 kernel: LustreError: 1118:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 19:57:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 19:57:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 20:02:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 20:02:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 20:02:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502161062, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800b264b600/0xf077f1a82c7d342a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8253085 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 20:02:42 oak-gw06 kernel: LustreError: 1166:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88038461fb40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 20:02:42 oak-gw06 kernel: LustreError: 1166:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 20:02:42 oak-gw06 kernel: LustreError: 1166:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88038461fb40) refcount = 2 Aug 7 20:02:42 oak-gw06 kernel: LustreError: 1166:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 20:02:42 oak-gw06 kernel: LustreError: 1166:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800b264b600/0xf077f1a82c7d342a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8253085 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 20:02:42 oak-gw06 kernel: LustreError: 1166:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 20:02:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 20:07:52 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 20:07:52 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 20:12:58 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 20:12:58 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 20:12:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502161678, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88028c347800/0xf077f1a82c7d727b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb826245f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 20:12:58 oak-gw06 kernel: LustreError: 1182:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88039ba3bb40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 20:12:58 oak-gw06 kernel: LustreError: 1182:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88039ba3bb40) refcount = 2 Aug 7 20:12:58 oak-gw06 kernel: LustreError: 1182:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 20:12:58 oak-gw06 kernel: LustreError: 1182:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88028c347800/0xf077f1a82c7d727b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb826245f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 20:12:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 20:18:07 oak-gw06 kernel: LustreError: 1186:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803212d1b40) refcount = 2 Aug 7 20:18:07 oak-gw06 kernel: LustreError: 1186:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 20:18:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 20:18:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 20:23:15 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 20:23:15 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 20:23:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502162295, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801f97ca000/0xf077f1a82c7d9dd2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8271b03 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 20:23:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 20:23:15 oak-gw06 kernel: LustreError: 1203:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802a3543cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 20:23:15 oak-gw06 kernel: LustreError: 1203:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 20:23:15 oak-gw06 kernel: LustreError: 1203:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a3543cc0) refcount = 2 Aug 7 20:23:15 oak-gw06 kernel: LustreError: 1203:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 20:23:15 oak-gw06 kernel: LustreError: 1203:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801f97ca000/0xf077f1a82c7d9dd2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8271b03 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 20:23:15 oak-gw06 kernel: LustreError: 1203:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 20:28:21 oak-gw06 kernel: LustreError: 1207:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88003aa65cc0) refcount = 2 Aug 7 20:28:21 oak-gw06 kernel: LustreError: 1207:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 20:28:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 20:28:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 20:33:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 20:33:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 20:33:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502162908, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88032ff70e00/0xf077f1a82c7dc95a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8280735 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 20:33:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 20:33:28 oak-gw06 kernel: LustreError: 1217:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800060e6e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 20:33:28 oak-gw06 kernel: LustreError: 1217:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 20:33:28 oak-gw06 kernel: LustreError: 1217:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800060e6e40) refcount = 2 Aug 7 20:33:28 oak-gw06 kernel: LustreError: 1217:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 20:33:28 oak-gw06 kernel: LustreError: 1217:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88032ff70e00/0xf077f1a82c7dc95a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8280735 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 20:33:28 oak-gw06 kernel: LustreError: 1217:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 20:38:37 oak-gw06 kernel: LustreError: 1227:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800060e6540) refcount = 2 Aug 7 20:38:37 oak-gw06 kernel: LustreError: 1227:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 20:38:37 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 20:38:37 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 20:43:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 20:43:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 20:43:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502163524, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800b9060e00/0xf077f1a82c7dffee lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb828fa60 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 20:43:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 20:43:44 oak-gw06 kernel: LustreError: 1240:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880043163300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 20:43:44 oak-gw06 kernel: LustreError: 1240:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 20:43:44 oak-gw06 kernel: LustreError: 1240:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880043163300) refcount = 2 Aug 7 20:43:44 oak-gw06 kernel: LustreError: 1240:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 20:43:44 oak-gw06 kernel: LustreError: 1240:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800b9060e00/0xf077f1a82c7dffee lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb828fa60 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 20:43:44 oak-gw06 kernel: LustreError: 1240:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 20:48:51 oak-gw06 kernel: LustreError: 1251:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800774f7600) refcount = 2 Aug 7 20:48:51 oak-gw06 kernel: LustreError: 1251:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 20:48:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 20:48:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 20:53:58 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 20:53:58 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 20:53:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502164138, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88022a2de200/0xf077f1a82c7e56c6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb829eb69 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 20:53:58 oak-gw06 kernel: LustreError: 1262:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880210797e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 20:53:58 oak-gw06 kernel: LustreError: 1262:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 20:53:58 oak-gw06 kernel: LustreError: 1262:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880210797e40) refcount = 2 Aug 7 20:53:58 oak-gw06 kernel: LustreError: 1262:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 20:53:58 oak-gw06 kernel: LustreError: 1262:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88022a2de200/0xf077f1a82c7e56c6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb829eb69 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 20:53:58 oak-gw06 kernel: LustreError: 1262:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 20:53:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 20:59:03 oak-gw06 kernel: LustreError: 1277:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88031a29e480) refcount = 2 Aug 7 20:59:03 oak-gw06 kernel: LustreError: 1277:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 20:59:03 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 20:59:03 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 21:04:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 21:04:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 21:04:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502164751, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801f997ce00/0xf077f1a82c7e6479 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb82ad835 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 21:04:11 oak-gw06 kernel: LustreError: 1320:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88034d76d240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 21:04:11 oak-gw06 kernel: LustreError: 1320:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 21:04:11 oak-gw06 kernel: LustreError: 1320:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88034d76d240) refcount = 2 Aug 7 21:04:11 oak-gw06 kernel: LustreError: 1320:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 21:04:11 oak-gw06 kernel: LustreError: 1320:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801f997ce00/0xf077f1a82c7e6479 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb82ad835 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 21:04:11 oak-gw06 kernel: LustreError: 1320:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 21:04:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 21:09:17 oak-gw06 kernel: LustreError: 1325:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880205badf00) refcount = 2 Aug 7 21:09:17 oak-gw06 kernel: LustreError: 1325:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 21:09:17 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 21:09:17 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 21:14:25 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 21:14:25 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 21:14:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502165365, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802e08a5000/0xf077f1a82c7e6e19 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb82bc586 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 21:14:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 21:14:25 oak-gw06 kernel: LustreError: 1335:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88042c7a3d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 21:14:25 oak-gw06 kernel: LustreError: 1335:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 21:14:25 oak-gw06 kernel: LustreError: 1335:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042c7a3d80) refcount = 2 Aug 7 21:14:25 oak-gw06 kernel: LustreError: 1335:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 21:14:25 oak-gw06 kernel: LustreError: 1335:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802e08a5000/0xf077f1a82c7e6e19 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb82bc586 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 21:14:25 oak-gw06 kernel: LustreError: 1335:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 21:19:33 oak-gw06 kernel: LustreError: 1340:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88023d35e900) refcount = 2 Aug 7 21:19:33 oak-gw06 kernel: LustreError: 1340:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 21:19:33 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 21:19:33 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 21:24:43 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 21:24:43 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 21:24:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502165983, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802027d9600/0xf077f1a82c7e775e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb82cbcf5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 21:24:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 21:24:43 oak-gw06 kernel: LustreError: 1351:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88023d35e3c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 21:24:43 oak-gw06 kernel: LustreError: 1351:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 21:24:43 oak-gw06 kernel: LustreError: 1351:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88023d35e3c0) refcount = 2 Aug 7 21:24:43 oak-gw06 kernel: LustreError: 1351:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 21:24:43 oak-gw06 kernel: LustreError: 1351:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802027d9600/0xf077f1a82c7e775e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb82cbcf5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 21:24:43 oak-gw06 kernel: LustreError: 1351:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 21:29:49 oak-gw06 kernel: LustreError: 1355:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802fae909c0) refcount = 2 Aug 7 21:29:49 oak-gw06 kernel: LustreError: 1355:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 21:29:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 21:29:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 21:34:58 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 21:34:58 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 21:34:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502166598, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88026b276e00/0xf077f1a82c7e7f06 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb82dad4f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 21:34:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 21:34:58 oak-gw06 kernel: LustreError: 1366:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802fae90900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 21:34:58 oak-gw06 kernel: LustreError: 1366:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 21:34:58 oak-gw06 kernel: LustreError: 1366:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802fae90900) refcount = 2 Aug 7 21:34:58 oak-gw06 kernel: LustreError: 1366:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 21:34:58 oak-gw06 kernel: LustreError: 1366:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88026b276e00/0xf077f1a82c7e7f06 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb82dad4f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 21:34:58 oak-gw06 kernel: LustreError: 1366:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 21:40:05 oak-gw06 kernel: LustreError: 1379:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8801c5b24c00) refcount = 2 Aug 7 21:40:05 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 21:40:05 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 21:45:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 21:45:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 21:45:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502167214, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801b9e87000/0xf077f1a82c7e848c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb82e9fd9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 21:45:14 oak-gw06 kernel: LustreError: 1381:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801b9e69900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 21:45:14 oak-gw06 kernel: LustreError: 1381:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 21:45:14 oak-gw06 kernel: LustreError: 1381:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801b9e69900) refcount = 2 Aug 7 21:45:14 oak-gw06 kernel: LustreError: 1381:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 21:45:14 oak-gw06 kernel: LustreError: 1381:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801b9e87000/0xf077f1a82c7e848c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb82e9fd9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 21:45:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 21:50:21 oak-gw06 kernel: LustreError: 1393:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041bc72000) refcount = 2 Aug 7 21:50:21 oak-gw06 kernel: LustreError: 1393:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 21:50:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 21:50:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 21:55:29 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 21:55:29 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 21:55:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502167829, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b8ded000/0xf077f1a82c7e8d37 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb82f908e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 21:55:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 21:55:29 oak-gw06 kernel: LustreError: 1398:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880009b54f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 21:55:29 oak-gw06 kernel: LustreError: 1398:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 21:55:29 oak-gw06 kernel: LustreError: 1398:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880009b54f00) refcount = 2 Aug 7 21:55:29 oak-gw06 kernel: LustreError: 1398:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 21:55:29 oak-gw06 kernel: LustreError: 1398:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b8ded000/0xf077f1a82c7e8d37 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb82f908e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 21:55:29 oak-gw06 kernel: LustreError: 1398:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 22:00:38 oak-gw06 kernel: LustreError: 1410:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880009b54d80) refcount = 2 Aug 7 22:00:38 oak-gw06 kernel: LustreError: 1410:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 22:00:38 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 22:00:38 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 22:05:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 22:05:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 22:05:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502168444, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88009fd2de00/0xf077f1a82c7e95b8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb83082a8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 22:05:44 oak-gw06 kernel: LustreError: 1452:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ecf7df00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 22:05:44 oak-gw06 kernel: LustreError: 1452:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 22:05:44 oak-gw06 kernel: LustreError: 1452:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ecf7df00) refcount = 2 Aug 7 22:05:44 oak-gw06 kernel: LustreError: 1452:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 22:05:44 oak-gw06 kernel: LustreError: 1452:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88009fd2de00/0xf077f1a82c7e95b8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb83082a8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 22:05:44 oak-gw06 kernel: LustreError: 1452:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 22:05:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 22:10:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 22:10:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 22:15:56 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 22:15:56 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 22:15:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502169056, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88028fe70800/0xf077f1a82c7e9e55 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8316caa expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 22:15:56 oak-gw06 kernel: LustreError: 1467:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88020f25d300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 22:15:56 oak-gw06 kernel: LustreError: 1467:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88020f25d300) refcount = 2 Aug 7 22:15:56 oak-gw06 kernel: LustreError: 1467:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 22:15:56 oak-gw06 kernel: LustreError: 1467:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88028fe70800/0xf077f1a82c7e9e55 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8316caa expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 22:15:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 22:21:04 oak-gw06 kernel: LustreError: 1479:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803f2625f00) refcount = 2 Aug 7 22:21:04 oak-gw06 kernel: LustreError: 1479:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 22:21:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 22:21:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 22:26:10 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 22:26:10 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 22:26:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502169670, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88001d51d200/0xf077f1a82c7ea6c8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8325b3d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 22:26:10 oak-gw06 kernel: LustreError: 1483:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880212f10f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 22:26:10 oak-gw06 kernel: LustreError: 1483:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 22:26:10 oak-gw06 kernel: LustreError: 1483:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880212f10f00) refcount = 2 Aug 7 22:26:10 oak-gw06 kernel: LustreError: 1483:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 22:26:10 oak-gw06 kernel: LustreError: 1483:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88001d51d200/0xf077f1a82c7ea6c8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8325b3d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 22:26:10 oak-gw06 kernel: LustreError: 1483:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 22:26:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 22:31:20 oak-gw06 kernel: LustreError: 1494:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802f669b240) refcount = 1 Aug 7 22:31:20 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 22:31:20 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 22:36:26 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 22:36:26 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 22:36:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502170286, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880051d2ca00/0xf077f1a82c7eae93 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8334c07 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 22:36:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 22:36:26 oak-gw06 kernel: LustreError: 1498:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880083db8300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 22:36:26 oak-gw06 kernel: LustreError: 1498:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 22:36:26 oak-gw06 kernel: LustreError: 1498:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880083db8300) refcount = 2 Aug 7 22:36:26 oak-gw06 kernel: LustreError: 1498:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 22:36:26 oak-gw06 kernel: LustreError: 1498:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880051d2ca00/0xf077f1a82c7eae93 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8334c07 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 22:41:32 oak-gw06 kernel: LustreError: 1514:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880083db8480) refcount = 2 Aug 7 22:41:32 oak-gw06 kernel: LustreError: 1514:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 22:41:32 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 22:41:32 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 22:46:39 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 22:46:39 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 22:46:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502170899, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88026b277200/0xf077f1a82c7eb792 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb83439ba expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 22:46:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 22:51:49 oak-gw06 kernel: LustreError: 1526:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801c5b24780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 22:51:49 oak-gw06 kernel: LustreError: 1526:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 22:51:49 oak-gw06 kernel: LustreError: 1526:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c5b24780) refcount = 2 Aug 7 22:51:49 oak-gw06 kernel: LustreError: 1526:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 22:51:49 oak-gw06 kernel: LustreError: 1526:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88026b276200/0xf077f1a82c7ebc2a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb834b3f4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 22:51:49 oak-gw06 kernel: LustreError: 1526:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 22:51:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 22:51:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 22:56:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 22:56:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 22:56:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502171519, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88026b276200/0xf077f1a82c7ebfb8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb83532d4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 22:56:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 22:56:59 oak-gw06 kernel: LustreError: 1529:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c5b24180) refcount = 2 Aug 7 22:56:59 oak-gw06 kernel: LustreError: 1529:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 23:02:09 oak-gw06 kernel: LustreError: 1576:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88042a983b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 23:02:09 oak-gw06 kernel: LustreError: 1576:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 23:02:09 oak-gw06 kernel: LustreError: 1576:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042a983b40) refcount = 2 Aug 7 23:02:09 oak-gw06 kernel: LustreError: 1576:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 23:02:09 oak-gw06 kernel: LustreError: 1576:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88026b276200/0xf077f1a82c7ec4c0 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb835add9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 23:02:09 oak-gw06 kernel: LustreError: 1576:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 23:02:09 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 23:02:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 23:07:19 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 23:07:19 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 23:07:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502172139, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88026b274400/0xf077f1a82c7eca2a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8362c81 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 23:07:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 23:07:19 oak-gw06 kernel: LustreError: 1580:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88019e141840) refcount = 2 Aug 7 23:07:19 oak-gw06 kernel: LustreError: 1580:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 23:12:26 oak-gw06 kernel: LustreError: 1592:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88019e1419c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 23:12:26 oak-gw06 kernel: LustreError: 1592:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 23:12:26 oak-gw06 kernel: LustreError: 1592:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88019e1419c0) refcount = 2 Aug 7 23:12:26 oak-gw06 kernel: LustreError: 1592:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 23:12:26 oak-gw06 kernel: LustreError: 1592:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88026b274400/0xf077f1a82c7ecd79 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb836a270 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 23:12:26 oak-gw06 kernel: LustreError: 1592:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 23:12:26 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 23:12:26 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 23:17:32 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 23:17:32 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 23:17:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502172752, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88026b274400/0xf077f1a82c7ed1e0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb83719ee expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 23:17:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 23:17:32 oak-gw06 kernel: LustreError: 1595:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88019e141cc0) refcount = 2 Aug 7 23:17:32 oak-gw06 kernel: LustreError: 1595:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 23:22:41 oak-gw06 kernel: LustreError: 1611:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880009b54d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 23:22:41 oak-gw06 kernel: LustreError: 1611:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 23:22:41 oak-gw06 kernel: LustreError: 1611:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880009b54d80) refcount = 2 Aug 7 23:22:41 oak-gw06 kernel: LustreError: 1611:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 23:22:41 oak-gw06 kernel: LustreError: 1611:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276f61e00/0xf077f1a82c7ed552 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8379380 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 23:22:41 oak-gw06 kernel: LustreError: 1611:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 23:22:41 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 23:22:41 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 23:27:50 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 23:27:50 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 23:27:50 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502173370, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276f60c00/0xf077f1a82c7ed949 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8381099 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 23:27:50 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 23:27:50 oak-gw06 kernel: LustreError: 1614:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880009b54780) refcount = 2 Aug 7 23:27:50 oak-gw06 kernel: LustreError: 1614:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 23:32:57 oak-gw06 kernel: LustreError: 1633:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880009b54000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 23:32:57 oak-gw06 kernel: LustreError: 1633:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 23:32:57 oak-gw06 kernel: LustreError: 1633:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880009b54000) refcount = 2 Aug 7 23:32:57 oak-gw06 kernel: LustreError: 1633:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 23:32:57 oak-gw06 kernel: LustreError: 1633:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276f63e00/0xf077f1a82c7edd01 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb838862d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 23:32:57 oak-gw06 kernel: LustreError: 1633:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 23:32:57 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 23:32:57 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 23:38:06 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 23:38:06 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 23:38:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502173986, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8804233eda00/0xf077f1a82c7ee22c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8390091 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 23:38:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 23:38:06 oak-gw06 kernel: LustreError: 1636:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88014f4efa80) refcount = 2 Aug 7 23:38:06 oak-gw06 kernel: LustreError: 1636:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 23:43:13 oak-gw06 kernel: LustreError: 1647:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88014f4ef3c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 23:43:13 oak-gw06 kernel: LustreError: 1647:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 23:43:13 oak-gw06 kernel: LustreError: 1647:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88014f4ef3c0) refcount = 2 Aug 7 23:43:13 oak-gw06 kernel: LustreError: 1647:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 23:43:13 oak-gw06 kernel: LustreError: 1647:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8804233edc00/0xf077f1a82c7ee64d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb839761e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 23:43:13 oak-gw06 kernel: LustreError: 1647:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 23:43:13 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 23:43:13 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 23:48:22 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 23:48:22 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 23:48:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502174602, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8804233edc00/0xf077f1a82c7ee9e9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb839f1ee expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 23:48:22 oak-gw06 kernel: LustreError: 1655:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88014f4ef480) refcount = 2 Aug 7 23:48:22 oak-gw06 kernel: LustreError: 1655:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 23:48:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 7 23:53:30 oak-gw06 kernel: LustreError: 1681:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88041bc723c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 7 23:53:30 oak-gw06 kernel: LustreError: 1681:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 7 23:53:30 oak-gw06 kernel: LustreError: 1681:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041bc723c0) refcount = 2 Aug 7 23:53:30 oak-gw06 kernel: LustreError: 1681:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 23:53:30 oak-gw06 kernel: LustreError: 1681:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8804233edc00/0xf077f1a82c7eee81 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb83a6903 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 23:53:30 oak-gw06 kernel: LustreError: 1681:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 7 23:53:30 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 7 23:53:30 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 7 23:58:36 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 7 23:58:36 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 7 23:58:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502175216, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880265b66a00/0xf077f1a82c7ef2a9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb83adf7e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 7 23:58:36 oak-gw06 kernel: LustreError: 1686:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880363f6e180) refcount = 2 Aug 7 23:58:36 oak-gw06 kernel: LustreError: 1686:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 7 23:58:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 00:03:42 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 00:03:42 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 00:08:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 00:08:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 00:08:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502175829, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880035a7c800/0xf077f1a82c7efb46 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb83bcc43 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 00:08:49 oak-gw06 kernel: LustreError: 1731:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88022a219240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 00:08:49 oak-gw06 kernel: LustreError: 1731:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 00:08:49 oak-gw06 kernel: LustreError: 1731:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022a219240) refcount = 2 Aug 8 00:08:49 oak-gw06 kernel: LustreError: 1731:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 00:08:49 oak-gw06 kernel: LustreError: 1731:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880035a7c800/0xf077f1a82c7efb46 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb83bcc43 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 00:08:49 oak-gw06 kernel: LustreError: 1731:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 00:08:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 00:13:55 oak-gw06 kernel: LustreError: 1752:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022a219540) refcount = 2 Aug 8 00:13:55 oak-gw06 kernel: LustreError: 1752:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 00:13:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 00:13:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 00:19:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 00:19:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 00:19:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502176442, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfaac00/0xf077f1a82c7f027e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb83cb6f4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 00:19:02 oak-gw06 kernel: LustreError: 1773:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88022a219d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 00:19:02 oak-gw06 kernel: LustreError: 1773:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 00:19:02 oak-gw06 kernel: LustreError: 1773:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022a219d80) refcount = 2 Aug 8 00:19:02 oak-gw06 kernel: LustreError: 1773:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 00:19:02 oak-gw06 kernel: LustreError: 1773:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfaac00/0xf077f1a82c7f027e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb83cb6f4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 00:19:02 oak-gw06 kernel: LustreError: 1773:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 00:19:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 00:24:08 oak-gw06 kernel: LustreError: 1795:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802d6e65180) refcount = 2 Aug 8 00:24:08 oak-gw06 kernel: LustreError: 1795:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 00:24:08 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 00:24:08 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 00:29:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 00:29:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 00:29:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502177054, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfab800/0xf077f1a82c7f0c25 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb83da32d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 00:29:14 oak-gw06 kernel: LustreError: 1799:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802d9db1e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 00:29:14 oak-gw06 kernel: LustreError: 1799:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 00:29:14 oak-gw06 kernel: LustreError: 1799:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802d9db1e40) refcount = 2 Aug 8 00:29:14 oak-gw06 kernel: LustreError: 1799:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 00:29:14 oak-gw06 kernel: LustreError: 1799:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfab800/0xf077f1a82c7f0c25 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb83da32d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 00:29:14 oak-gw06 kernel: LustreError: 1799:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 00:29:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 00:34:23 oak-gw06 kernel: LustreError: 1811:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802d9db1780) refcount = 2 Aug 8 00:34:23 oak-gw06 kernel: LustreError: 1811:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 00:34:23 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 00:34:23 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 00:39:29 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 00:39:29 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 00:39:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502177669, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88039dfa6800/0xf077f1a82c7f1460 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb83e91c7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 00:39:29 oak-gw06 kernel: LustreError: 1815:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803bf3af240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 00:39:29 oak-gw06 kernel: LustreError: 1815:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 00:39:29 oak-gw06 kernel: LustreError: 1815:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803bf3af240) refcount = 2 Aug 8 00:39:29 oak-gw06 kernel: LustreError: 1815:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 00:39:29 oak-gw06 kernel: LustreError: 1815:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88039dfa6800/0xf077f1a82c7f1460 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb83e91c7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 00:39:29 oak-gw06 kernel: LustreError: 1815:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 00:39:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 00:44:38 oak-gw06 kernel: LustreError: 1828:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880295b77900) refcount = 2 Aug 8 00:44:38 oak-gw06 kernel: LustreError: 1828:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 00:44:38 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 00:44:38 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 00:49:46 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 00:49:46 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 00:49:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502178286, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803ecb6e200/0xf077f1a82c7f1ce8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb83f86d5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 00:49:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 00:49:46 oak-gw06 kernel: LustreError: 1830:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880295b77600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 00:49:46 oak-gw06 kernel: LustreError: 1830:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 00:49:46 oak-gw06 kernel: LustreError: 1830:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880295b77600) refcount = 2 Aug 8 00:49:46 oak-gw06 kernel: LustreError: 1830:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 00:49:46 oak-gw06 kernel: LustreError: 1830:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803ecb6e200/0xf077f1a82c7f1ce8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb83f86d5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 00:49:46 oak-gw06 kernel: LustreError: 1830:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 00:54:52 oak-gw06 kernel: LustreError: 1840:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802d6e65000) refcount = 2 Aug 8 00:54:52 oak-gw06 kernel: LustreError: 1840:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 00:54:52 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 00:54:52 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 00:59:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 00:59:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 00:59:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502178899, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803ecb6e600/0xf077f1a82c7f2489 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb84074b9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 00:59:59 oak-gw06 kernel: LustreError: 1845:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802d6e659c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 00:59:59 oak-gw06 kernel: LustreError: 1845:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 00:59:59 oak-gw06 kernel: LustreError: 1845:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802d6e659c0) refcount = 2 Aug 8 00:59:59 oak-gw06 kernel: LustreError: 1845:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 00:59:59 oak-gw06 kernel: LustreError: 1845:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803ecb6e600/0xf077f1a82c7f2489 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb84074b9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 00:59:59 oak-gw06 kernel: LustreError: 1845:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 00:59:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 01:05:06 oak-gw06 kernel: LustreError: 1893:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802d6e65f00) refcount = 2 Aug 8 01:05:06 oak-gw06 kernel: LustreError: 1893:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 01:05:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 01:05:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 01:10:12 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 01:10:12 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 01:10:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502179512, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880035a7f400/0xf077f1a82c7f2c8c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8416265 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 01:10:12 oak-gw06 kernel: LustreError: 1904:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880212f10a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 01:10:12 oak-gw06 kernel: LustreError: 1904:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 01:10:12 oak-gw06 kernel: LustreError: 1904:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880212f10a80) refcount = 2 Aug 8 01:10:12 oak-gw06 kernel: LustreError: 1904:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 01:10:12 oak-gw06 kernel: LustreError: 1904:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880035a7f400/0xf077f1a82c7f2c8c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8416265 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 01:10:12 oak-gw06 kernel: LustreError: 1904:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 01:10:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 01:15:21 oak-gw06 kernel: LustreError: 1908:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880212f10e40) refcount = 2 Aug 8 01:15:21 oak-gw06 kernel: LustreError: 1908:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 01:15:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 01:15:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 01:20:29 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 01:20:29 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 01:20:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502180129, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880035a7f400/0xf077f1a82c7f3450 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb842569a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 01:20:29 oak-gw06 kernel: LustreError: 1921:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880212f10840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 01:20:29 oak-gw06 kernel: LustreError: 1921:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 01:20:29 oak-gw06 kernel: LustreError: 1921:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880212f10840) refcount = 2 Aug 8 01:20:29 oak-gw06 kernel: LustreError: 1921:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 01:20:29 oak-gw06 kernel: LustreError: 1921:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880035a7f400/0xf077f1a82c7f3450 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb842569a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 01:20:29 oak-gw06 kernel: LustreError: 1921:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 01:20:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 01:25:36 oak-gw06 kernel: LustreError: 1927:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d37c780) refcount = 2 Aug 8 01:25:36 oak-gw06 kernel: LustreError: 1927:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 01:25:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 01:25:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 01:30:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 01:30:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 01:30:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502180741, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880147360200/0xf077f1a82c7f3cdf lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8434423 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 01:30:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 01:30:41 oak-gw06 kernel: LustreError: 1942:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800b9959840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 01:30:41 oak-gw06 kernel: LustreError: 1942:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 01:30:41 oak-gw06 kernel: LustreError: 1942:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800b9959840) refcount = 2 Aug 8 01:30:41 oak-gw06 kernel: LustreError: 1942:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 01:30:41 oak-gw06 kernel: LustreError: 1942:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880147360200/0xf077f1a82c7f3cdf lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8434423 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 01:30:41 oak-gw06 kernel: LustreError: 1942:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 01:35:48 oak-gw06 kernel: LustreError: 1947:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8800b9959b40) refcount = 2 Aug 8 01:35:48 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 01:35:48 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 01:40:56 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 01:40:56 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 01:40:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502181356, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276f63e00/0xf077f1a82c7f469b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb84433ab expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 01:40:56 oak-gw06 kernel: LustreError: 1956:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800b99599c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 01:40:56 oak-gw06 kernel: LustreError: 1956:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 01:40:56 oak-gw06 kernel: LustreError: 1956:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800b99599c0) refcount = 2 Aug 8 01:40:56 oak-gw06 kernel: LustreError: 1956:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 01:40:56 oak-gw06 kernel: LustreError: 1956:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276f63e00/0xf077f1a82c7f469b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb84433ab expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 01:40:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 01:46:02 oak-gw06 kernel: LustreError: 1959:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bb6410c0) refcount = 2 Aug 8 01:46:02 oak-gw06 kernel: LustreError: 1959:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 01:46:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 01:46:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 01:51:08 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 01:51:08 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 01:51:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502181968, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880051d2c800/0xf077f1a82c7f4f1c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8452188 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 01:51:08 oak-gw06 kernel: LustreError: 1970:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803bf3af840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 01:51:08 oak-gw06 kernel: LustreError: 1970:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 01:51:08 oak-gw06 kernel: LustreError: 1970:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803bf3af840) refcount = 2 Aug 8 01:51:08 oak-gw06 kernel: LustreError: 1970:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 01:51:08 oak-gw06 kernel: LustreError: 1970:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880051d2c800/0xf077f1a82c7f4f1c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8452188 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 01:51:08 oak-gw06 kernel: LustreError: 1970:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 01:51:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 01:56:16 oak-gw06 kernel: LustreError: 1974:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802912e0780) refcount = 2 Aug 8 01:56:16 oak-gw06 kernel: LustreError: 1974:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 01:56:16 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 01:56:16 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 02:01:22 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 02:01:22 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 02:01:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502182582, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfaa000/0xf077f1a82c7f571f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb84610f4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 02:01:22 oak-gw06 kernel: LustreError: 2130:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802d9db1b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 02:01:22 oak-gw06 kernel: LustreError: 2130:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 02:01:22 oak-gw06 kernel: LustreError: 2130:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802d9db1b40) refcount = 2 Aug 8 02:01:22 oak-gw06 kernel: LustreError: 2130:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 02:01:22 oak-gw06 kernel: LustreError: 2130:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfaa000/0xf077f1a82c7f571f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb84610f4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 02:01:22 oak-gw06 kernel: LustreError: 2130:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 02:01:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 02:06:32 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 02:06:32 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 02:11:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 02:11:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 02:11:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502183202, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfaa000/0xf077f1a82c7f6025 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb84706a3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 02:11:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 02:11:42 oak-gw06 kernel: LustreError: 2142:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880413f396c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 02:11:42 oak-gw06 kernel: LustreError: 2142:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413f396c0) refcount = 2 Aug 8 02:11:42 oak-gw06 kernel: LustreError: 2142:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 02:11:42 oak-gw06 kernel: LustreError: 2142:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfaa000/0xf077f1a82c7f6025 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb84706a3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 02:16:51 oak-gw06 kernel: LustreError: 2145:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413f39e40) refcount = 2 Aug 8 02:16:51 oak-gw06 kernel: LustreError: 2145:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 02:16:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 02:16:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 02:21:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 02:21:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 02:21:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502183817, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfaa000/0xf077f1a82c7f66f4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb847f5f3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 02:21:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 02:21:57 oak-gw06 kernel: LustreError: 2155:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880413f39840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 02:21:57 oak-gw06 kernel: LustreError: 2155:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 02:21:57 oak-gw06 kernel: LustreError: 2155:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413f39840) refcount = 2 Aug 8 02:21:57 oak-gw06 kernel: LustreError: 2155:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 02:21:57 oak-gw06 kernel: LustreError: 2155:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfaa000/0xf077f1a82c7f66f4 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb847f5f3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 02:21:57 oak-gw06 kernel: LustreError: 2155:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 02:27:05 oak-gw06 kernel: LustreError: 2160:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042c4bf480) refcount = 2 Aug 8 02:27:05 oak-gw06 kernel: LustreError: 2160:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 02:27:05 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 02:27:05 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 02:32:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 02:32:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 02:32:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502184433, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880035a7de00/0xf077f1a82c7f6f83 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb848e86f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 02:32:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 02:32:13 oak-gw06 kernel: LustreError: 2171:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880212f10180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 02:32:13 oak-gw06 kernel: LustreError: 2171:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 02:32:13 oak-gw06 kernel: LustreError: 2171:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880212f10180) refcount = 2 Aug 8 02:32:13 oak-gw06 kernel: LustreError: 2171:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 02:32:13 oak-gw06 kernel: LustreError: 2171:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880035a7de00/0xf077f1a82c7f6f83 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb848e86f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 02:32:13 oak-gw06 kernel: LustreError: 2171:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 02:37:23 oak-gw06 kernel: LustreError: 2175:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c1ff6540) refcount = 2 Aug 8 02:37:23 oak-gw06 kernel: LustreError: 2175:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 02:37:23 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 02:37:23 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 02:42:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 02:42:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 02:42:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502185051, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880147360600/0xf077f1a82c7f77b7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb849dc42 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 02:42:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 02:42:31 oak-gw06 kernel: LustreError: 2188:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801e63d4840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 02:42:31 oak-gw06 kernel: LustreError: 2188:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 02:42:31 oak-gw06 kernel: LustreError: 2188:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801e63d4840) refcount = 2 Aug 8 02:42:31 oak-gw06 kernel: LustreError: 2188:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 02:42:31 oak-gw06 kernel: LustreError: 2188:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880147360600/0xf077f1a82c7f77b7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb849dc42 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 02:42:31 oak-gw06 kernel: LustreError: 2188:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 02:47:36 oak-gw06 kernel: LustreError: 2192:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413fac300) refcount = 2 Aug 8 02:47:36 oak-gw06 kernel: LustreError: 2192:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 02:47:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 02:47:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 02:52:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 02:52:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 02:52:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502185662, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88035e86f800/0xf077f1a82c7f7fac lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb84ac8c8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 02:52:42 oak-gw06 kernel: LustreError: 2203:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801e63d4cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 02:52:42 oak-gw06 kernel: LustreError: 2203:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 02:52:42 oak-gw06 kernel: LustreError: 2203:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801e63d4cc0) refcount = 2 Aug 8 02:52:42 oak-gw06 kernel: LustreError: 2203:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 02:52:42 oak-gw06 kernel: LustreError: 2203:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88035e86f800/0xf077f1a82c7f7fac lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb84ac8c8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 02:52:42 oak-gw06 kernel: LustreError: 2203:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 02:52:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 02:57:49 oak-gw06 kernel: LustreError: 2207:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041d5d40c0) refcount = 2 Aug 8 02:57:49 oak-gw06 kernel: LustreError: 2207:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 02:57:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 02:57:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 03:02:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 03:02:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 03:02:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502186279, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880075955400/0xf077f1a82c7f8803 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb84bbb67 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 03:02:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 03:02:59 oak-gw06 kernel: LustreError: 2249:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880295b77840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 03:02:59 oak-gw06 kernel: LustreError: 2249:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 03:02:59 oak-gw06 kernel: LustreError: 2249:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880295b77840) refcount = 2 Aug 8 03:02:59 oak-gw06 kernel: LustreError: 2249:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 03:02:59 oak-gw06 kernel: LustreError: 2249:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880075955400/0xf077f1a82c7f8803 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb84bbb67 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 03:02:59 oak-gw06 kernel: LustreError: 2249:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 03:08:08 oak-gw06 kernel: LustreError: 2253:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801434dd3c0) refcount = 2 Aug 8 03:08:08 oak-gw06 kernel: LustreError: 2253:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 03:08:08 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 03:08:08 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 03:13:17 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 03:13:17 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 03:13:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502186897, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880075957a00/0xf077f1a82c7f91db lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb84caffe expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 03:13:17 oak-gw06 kernel: LustreError: 2263:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801434dd9c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 03:13:17 oak-gw06 kernel: LustreError: 2263:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 03:13:17 oak-gw06 kernel: LustreError: 2263:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801434dd9c0) refcount = 2 Aug 8 03:13:17 oak-gw06 kernel: LustreError: 2263:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 03:13:17 oak-gw06 kernel: LustreError: 2263:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880075957a00/0xf077f1a82c7f91db lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb84caffe expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 03:13:17 oak-gw06 kernel: LustreError: 2263:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 03:13:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 03:18:24 oak-gw06 kernel: LustreError: 2271:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041be659c0) refcount = 2 Aug 8 03:18:24 oak-gw06 kernel: LustreError: 2271:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 03:18:24 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 03:18:24 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 03:23:34 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 03:23:34 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 03:23:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502187514, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880035a7f000/0xf077f1a82c7f9a9b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb84da45d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 03:23:34 oak-gw06 kernel: LustreError: 2280:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88021070fc00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 03:23:34 oak-gw06 kernel: LustreError: 2280:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 03:23:34 oak-gw06 kernel: LustreError: 2280:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88021070fc00) refcount = 2 Aug 8 03:23:34 oak-gw06 kernel: LustreError: 2280:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 03:23:34 oak-gw06 kernel: LustreError: 2280:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880035a7f000/0xf077f1a82c7f9a9b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb84da45d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 03:23:34 oak-gw06 kernel: LustreError: 2280:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 03:23:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 03:28:39 oak-gw06 kernel: LustreError: 2284:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880418abc900) refcount = 2 Aug 8 03:28:39 oak-gw06 kernel: LustreError: 2284:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 03:28:39 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 03:28:39 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 03:33:46 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 03:33:46 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 03:33:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502188126, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfa8a00/0xf077f1a82c7fa2e4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb84e9295 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 03:33:46 oak-gw06 kernel: LustreError: 2294:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880090f8ea80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 03:33:46 oak-gw06 kernel: LustreError: 2294:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 03:33:46 oak-gw06 kernel: LustreError: 2294:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880090f8ea80) refcount = 2 Aug 8 03:33:46 oak-gw06 kernel: LustreError: 2294:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 03:33:46 oak-gw06 kernel: LustreError: 2294:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfa8a00/0xf077f1a82c7fa2e4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb84e9295 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 03:33:46 oak-gw06 kernel: LustreError: 2294:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 03:33:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 03:38:53 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 03:38:53 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 03:44:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 03:44:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 03:44:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502188742, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfab800/0xf077f1a82c7fa97b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb84f85ab expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 03:44:02 oak-gw06 kernel: LustreError: 2328:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880090f8e180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 03:44:02 oak-gw06 kernel: LustreError: 2328:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880090f8e180) refcount = 2 Aug 8 03:44:02 oak-gw06 kernel: LustreError: 2328:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 03:44:02 oak-gw06 kernel: LustreError: 2328:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfab800/0xf077f1a82c7fa97b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb84f85ab expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 03:44:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 03:49:07 oak-gw06 kernel: LustreError: 2332:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880090f8e3c0) refcount = 2 Aug 8 03:49:07 oak-gw06 kernel: LustreError: 2332:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 03:49:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 03:49:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 03:54:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 03:54:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 03:54:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502189354, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfaa200/0xf077f1a82c7fb2e3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8507541 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 03:54:14 oak-gw06 kernel: LustreError: 2342:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802a3543b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 03:54:14 oak-gw06 kernel: LustreError: 2342:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 03:54:14 oak-gw06 kernel: LustreError: 2342:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a3543b40) refcount = 2 Aug 8 03:54:14 oak-gw06 kernel: LustreError: 2342:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 03:54:14 oak-gw06 kernel: LustreError: 2342:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfaa200/0xf077f1a82c7fb2e3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8507541 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 03:54:14 oak-gw06 kernel: LustreError: 2342:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 03:54:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 03:59:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 03:59:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 04:04:27 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 04:04:27 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 04:04:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502189967, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfab200/0xf077f1a82c7fbce5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb85164e5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 04:04:27 oak-gw06 kernel: LustreError: 2386:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802a3543840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 04:04:27 oak-gw06 kernel: LustreError: 2386:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a3543840) refcount = 2 Aug 8 04:04:27 oak-gw06 kernel: LustreError: 2386:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 04:04:27 oak-gw06 kernel: LustreError: 2386:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88005dfab200/0xf077f1a82c7fbce5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb85164e5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 04:04:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 04:09:34 oak-gw06 kernel: LustreError: 2390:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880090f8e600) refcount = 2 Aug 8 04:09:34 oak-gw06 kernel: LustreError: 2390:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 04:09:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 04:09:34 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 04:14:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 04:14:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 04:14:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502190584, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802e358d000/0xf077f1a82c7fc4e8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8525864 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 04:14:44 oak-gw06 kernel: LustreError: 2403:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88028bf106c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 04:14:44 oak-gw06 kernel: LustreError: 2403:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 04:14:44 oak-gw06 kernel: LustreError: 2403:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88028bf106c0) refcount = 2 Aug 8 04:14:44 oak-gw06 kernel: LustreError: 2403:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 04:14:44 oak-gw06 kernel: LustreError: 2403:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802e358d000/0xf077f1a82c7fc4e8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8525864 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 04:14:44 oak-gw06 kernel: LustreError: 2403:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 04:14:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 04:19:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 04:19:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 04:24:58 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 04:24:58 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 04:24:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502191198, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802e358f200/0xf077f1a82c7fcd5b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb853489b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 04:24:58 oak-gw06 kernel: LustreError: 2418:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88028bf10c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 04:24:58 oak-gw06 kernel: LustreError: 2418:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88028bf10c00) refcount = 2 Aug 8 04:24:58 oak-gw06 kernel: LustreError: 2418:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 04:24:58 oak-gw06 kernel: LustreError: 2418:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802e358f200/0xf077f1a82c7fcd5b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb853489b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 04:24:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 04:30:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 04:30:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 04:35:12 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 04:35:12 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 04:35:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502191812, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880075957000/0xf077f1a82c7fdfd0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb85439e3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 04:35:12 oak-gw06 kernel: LustreError: 2432:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880418abcd80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 04:35:12 oak-gw06 kernel: LustreError: 2432:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880418abcd80) refcount = 2 Aug 8 04:35:12 oak-gw06 kernel: LustreError: 2432:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 04:35:12 oak-gw06 kernel: LustreError: 2432:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880075957000/0xf077f1a82c7fdfd0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb85439e3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 04:35:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 04:40:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 04:40:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 04:45:29 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 04:45:29 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 04:45:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502192429, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880357aec600/0xf077f1a82c81ff06 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8552cdd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 04:45:29 oak-gw06 kernel: LustreError: 2475:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803af3ea300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 04:45:29 oak-gw06 kernel: LustreError: 2475:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803af3ea300) refcount = 2 Aug 8 04:45:29 oak-gw06 kernel: LustreError: 2475:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 04:45:29 oak-gw06 kernel: LustreError: 2475:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880357aec600/0xf077f1a82c81ff06 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8552cdd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 04:45:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 04:50:35 oak-gw06 kernel: LustreError: 2485:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803af3eacc0) refcount = 2 Aug 8 04:50:35 oak-gw06 kernel: LustreError: 2485:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 04:50:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 04:50:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 04:55:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 04:55:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 04:55:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502193044, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88028fe71800/0xf077f1a82c822842 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8561dd1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 04:55:44 oak-gw06 kernel: LustreError: 2497:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880157fc90c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 04:55:44 oak-gw06 kernel: LustreError: 2497:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 04:55:44 oak-gw06 kernel: LustreError: 2497:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880157fc90c0) refcount = 2 Aug 8 04:55:44 oak-gw06 kernel: LustreError: 2497:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 04:55:44 oak-gw06 kernel: LustreError: 2497:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88028fe71800/0xf077f1a82c822842 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8561dd1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 04:55:44 oak-gw06 kernel: LustreError: 2497:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 04:55:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 05:00:51 oak-gw06 kernel: LustreError: 2508:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88028fa0c6c0) refcount = 2 Aug 8 05:00:51 oak-gw06 kernel: LustreError: 2508:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 05:00:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 05:00:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 05:06:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 05:06:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 05:06:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502193660, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88003bf40600/0xf077f1a82c82cf9a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8570f12 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 05:06:00 oak-gw06 kernel: LustreError: 2549:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802037cc840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 05:06:00 oak-gw06 kernel: LustreError: 2549:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 05:06:00 oak-gw06 kernel: LustreError: 2549:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802037cc840) refcount = 2 Aug 8 05:06:00 oak-gw06 kernel: LustreError: 2549:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 05:06:00 oak-gw06 kernel: LustreError: 2549:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88003bf40600/0xf077f1a82c82cf9a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8570f12 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 05:06:00 oak-gw06 kernel: LustreError: 2549:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 05:06:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 05:11:06 oak-gw06 kernel: LustreError: 2561:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801843b2300) refcount = 1 Aug 8 05:11:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 05:11:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 05:16:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 05:16:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 05:16:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502194273, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88004dc4f800/0xf077f1a82c82f938 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb857fdf9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 05:16:13 oak-gw06 kernel: LustreError: 2562:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a2f75480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 05:16:13 oak-gw06 kernel: LustreError: 2562:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 05:16:13 oak-gw06 kernel: LustreError: 2562:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a2f75480) refcount = 2 Aug 8 05:16:13 oak-gw06 kernel: LustreError: 2562:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 05:16:13 oak-gw06 kernel: LustreError: 2562:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88004dc4f800/0xf077f1a82c82f938 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb857fdf9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 05:16:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 05:21:22 oak-gw06 kernel: LustreError: 2578:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880099e94d80) refcount = 2 Aug 8 05:21:22 oak-gw06 kernel: LustreError: 2578:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 05:21:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 05:21:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 05:26:32 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 05:26:32 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 05:26:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502194892, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802173c1e00/0xf077f1a82c834582 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb858f17f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 05:26:32 oak-gw06 kernel: LustreError: 2582:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801b2c67540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 05:26:32 oak-gw06 kernel: LustreError: 2582:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 05:26:32 oak-gw06 kernel: LustreError: 2582:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801b2c67540) refcount = 2 Aug 8 05:26:32 oak-gw06 kernel: LustreError: 2582:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 05:26:32 oak-gw06 kernel: LustreError: 2582:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802173c1e00/0xf077f1a82c834582 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb858f17f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 05:26:32 oak-gw06 kernel: LustreError: 2582:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 05:26:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 05:31:42 oak-gw06 kernel: LustreError: 2598:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803f0eb0c00) refcount = 2 Aug 8 05:31:42 oak-gw06 kernel: LustreError: 2598:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 05:31:42 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 05:31:42 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 05:36:48 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 05:36:48 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 05:36:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502195508, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880240618200/0xf077f1a82c837e85 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb859e399 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 05:36:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 05:36:48 oak-gw06 kernel: LustreError: 2602:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880235323900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 05:36:48 oak-gw06 kernel: LustreError: 2602:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 05:36:48 oak-gw06 kernel: LustreError: 2602:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880235323900) refcount = 2 Aug 8 05:36:48 oak-gw06 kernel: LustreError: 2602:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 05:36:48 oak-gw06 kernel: LustreError: 2602:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880240618200/0xf077f1a82c837e85 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb859e399 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 05:36:48 oak-gw06 kernel: LustreError: 2602:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 05:41:54 oak-gw06 kernel: LustreError: 2613:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880293b51f00) refcount = 2 Aug 8 05:41:54 oak-gw06 kernel: LustreError: 2613:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 05:41:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 05:41:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 05:47:01 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 05:47:01 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 05:47:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502196121, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801d73e8000/0xf077f1a82c83a273 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb85ad2fe expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 05:47:01 oak-gw06 kernel: LustreError: 2617:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880378b0a240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 05:47:01 oak-gw06 kernel: LustreError: 2617:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 05:47:01 oak-gw06 kernel: LustreError: 2617:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880378b0a240) refcount = 2 Aug 8 05:47:01 oak-gw06 kernel: LustreError: 2617:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 05:47:01 oak-gw06 kernel: LustreError: 2617:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801d73e8000/0xf077f1a82c83a273 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb85ad2fe expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 05:47:01 oak-gw06 kernel: LustreError: 2617:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 05:47:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 05:52:07 oak-gw06 kernel: LustreError: 2633:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ec24a300) refcount = 2 Aug 8 05:52:07 oak-gw06 kernel: LustreError: 2633:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 05:52:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 05:52:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 05:57:17 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 05:57:17 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 05:57:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502196737, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880209f85400/0xf077f1a82c83c8bb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb85bc43f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 05:57:17 oak-gw06 kernel: LustreError: 2636:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88030d830a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 05:57:17 oak-gw06 kernel: LustreError: 2636:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 05:57:17 oak-gw06 kernel: LustreError: 2636:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88030d830a80) refcount = 2 Aug 8 05:57:17 oak-gw06 kernel: LustreError: 2636:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 05:57:17 oak-gw06 kernel: LustreError: 2636:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880209f85400/0xf077f1a82c83c8bb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb85bc43f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 05:57:17 oak-gw06 kernel: LustreError: 2636:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 05:57:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 06:02:26 oak-gw06 kernel: LustreError: 2681:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803dd2d8780) refcount = 2 Aug 8 06:02:26 oak-gw06 kernel: LustreError: 2681:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 06:02:26 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 06:02:26 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 06:07:33 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 06:07:33 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 06:07:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502197353, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880004628200/0xf077f1a82c83f4c8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb85cb80b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 06:07:33 oak-gw06 kernel: LustreError: 2689:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88028a288180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 06:07:33 oak-gw06 kernel: LustreError: 2689:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 06:07:33 oak-gw06 kernel: LustreError: 2689:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88028a288180) refcount = 2 Aug 8 06:07:33 oak-gw06 kernel: LustreError: 2689:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 06:07:33 oak-gw06 kernel: LustreError: 2689:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880004628200/0xf077f1a82c83f4c8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb85cb80b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 06:07:33 oak-gw06 kernel: LustreError: 2689:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 06:07:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 06:12:40 oak-gw06 kernel: LustreError: 2699:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803b227fcc0) refcount = 2 Aug 8 06:12:40 oak-gw06 kernel: LustreError: 2699:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 06:12:40 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 06:12:40 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 06:17:47 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 06:17:47 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 06:17:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502197967, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880196995600/0xf077f1a82c842b1d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb85da7f5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 06:17:47 oak-gw06 kernel: LustreError: 2703:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801a1f8a0c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 06:17:47 oak-gw06 kernel: LustreError: 2703:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 06:17:47 oak-gw06 kernel: LustreError: 2703:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801a1f8a0c0) refcount = 2 Aug 8 06:17:47 oak-gw06 kernel: LustreError: 2703:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 06:17:47 oak-gw06 kernel: LustreError: 2703:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880196995600/0xf077f1a82c842b1d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb85da7f5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 06:17:47 oak-gw06 kernel: LustreError: 2703:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 06:17:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 06:22:54 oak-gw06 kernel: LustreError: 2718:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802037cce40) refcount = 2 Aug 8 06:22:54 oak-gw06 kernel: LustreError: 2718:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 06:22:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 06:22:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 06:28:04 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 06:28:04 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 06:28:04 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502198584, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88021b3f6000/0xf077f1a82c84711d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb85e9a9b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 06:28:04 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 06:28:04 oak-gw06 kernel: LustreError: 2722:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88021451dc00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 06:28:04 oak-gw06 kernel: LustreError: 2722:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 06:28:04 oak-gw06 kernel: LustreError: 2722:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88021451dc00) refcount = 2 Aug 8 06:28:04 oak-gw06 kernel: LustreError: 2722:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 06:28:04 oak-gw06 kernel: LustreError: 2722:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88021b3f6000/0xf077f1a82c84711d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb85e9a9b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 06:28:04 oak-gw06 kernel: LustreError: 2722:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 06:33:14 oak-gw06 kernel: LustreError: 2737:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88021451d0c0) refcount = 1 Aug 8 06:33:14 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 06:33:14 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 06:38:23 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 06:38:23 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 06:38:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502199203, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880378e12000/0xf077f1a82c84beef lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb85f8fcc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 06:38:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 06:38:23 oak-gw06 kernel: LustreError: 2739:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802546d9b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 06:38:23 oak-gw06 kernel: LustreError: 2739:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 06:38:23 oak-gw06 kernel: LustreError: 2739:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802546d9b40) refcount = 2 Aug 8 06:38:23 oak-gw06 kernel: LustreError: 2739:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 06:38:23 oak-gw06 kernel: LustreError: 2739:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880378e12000/0xf077f1a82c84beef lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb85f8fcc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 06:43:29 oak-gw06 kernel: LustreError: 2754:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88035e977a80) refcount = 2 Aug 8 06:43:29 oak-gw06 kernel: LustreError: 2754:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 06:43:29 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 06:43:29 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 06:48:36 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 06:48:36 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 06:48:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502199816, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880049d1d400/0xf077f1a82c85308c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8608034 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 06:48:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 06:48:36 oak-gw06 kernel: LustreError: 2761:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880361f85cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 06:48:36 oak-gw06 kernel: LustreError: 2761:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 06:48:36 oak-gw06 kernel: LustreError: 2761:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880361f85cc0) refcount = 2 Aug 8 06:48:36 oak-gw06 kernel: LustreError: 2761:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 06:48:36 oak-gw06 kernel: LustreError: 2761:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880049d1d400/0xf077f1a82c85308c lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8608034 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 06:48:36 oak-gw06 kernel: LustreError: 2761:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 06:53:46 oak-gw06 kernel: LustreError: 2772:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880399dce9c0) refcount = 2 Aug 8 06:53:46 oak-gw06 kernel: LustreError: 2772:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 06:53:46 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 06:53:46 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 06:58:51 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 06:58:51 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 06:58:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502200431, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880157eb5c00/0xf077f1a82c859dc9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8617247 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 06:58:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 06:58:51 oak-gw06 kernel: LustreError: 2775:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802dcfc6840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 06:58:51 oak-gw06 kernel: LustreError: 2775:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 06:58:51 oak-gw06 kernel: LustreError: 2775:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802dcfc6840) refcount = 2 Aug 8 06:58:51 oak-gw06 kernel: LustreError: 2775:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 06:58:51 oak-gw06 kernel: LustreError: 2775:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880157eb5c00/0xf077f1a82c859dc9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8617247 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 06:58:51 oak-gw06 kernel: LustreError: 2775:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 07:04:01 oak-gw06 kernel: LustreError: 2823:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803097bb9c0) refcount = 2 Aug 8 07:04:01 oak-gw06 kernel: LustreError: 2823:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 07:04:01 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 07:04:01 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 07:09:07 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 07:09:07 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 07:09:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502201047, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801baed8a00/0xf077f1a82c85db56 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb86264ca expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 07:09:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 07:09:07 oak-gw06 kernel: LustreError: 2827:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880206bf5e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 07:09:07 oak-gw06 kernel: LustreError: 2827:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 07:09:07 oak-gw06 kernel: LustreError: 2827:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880206bf5e40) refcount = 2 Aug 8 07:09:07 oak-gw06 kernel: LustreError: 2827:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 07:09:07 oak-gw06 kernel: LustreError: 2827:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801baed8a00/0xf077f1a82c85db56 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb86264ca expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 07:09:07 oak-gw06 kernel: LustreError: 2827:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 07:14:17 oak-gw06 kernel: LustreError: 2839:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803deec63c0) refcount = 2 Aug 8 07:14:17 oak-gw06 kernel: LustreError: 2839:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 07:14:17 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 07:14:17 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 07:19:27 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 07:19:27 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 07:19:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502201667, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880041404400/0xf077f1a82c860c09 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb863593e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 07:19:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 07:19:27 oak-gw06 kernel: LustreError: 2842:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88000ad54780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 07:19:27 oak-gw06 kernel: LustreError: 2842:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 07:19:27 oak-gw06 kernel: LustreError: 2842:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88000ad54780) refcount = 2 Aug 8 07:19:27 oak-gw06 kernel: LustreError: 2842:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 07:19:27 oak-gw06 kernel: LustreError: 2842:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880041404400/0xf077f1a82c860c09 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb863593e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 07:19:27 oak-gw06 kernel: LustreError: 2842:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 07:24:32 oak-gw06 kernel: LustreError: 2853:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042e9bc300) refcount = 2 Aug 8 07:24:32 oak-gw06 kernel: LustreError: 2853:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 07:24:32 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 07:24:32 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 07:29:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 07:29:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 07:29:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502202281, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880041404e00/0xf077f1a82c860c48 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb86449de expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 07:29:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 07:29:41 oak-gw06 kernel: LustreError: 2857:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88003d58e540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 07:29:41 oak-gw06 kernel: LustreError: 2857:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 07:29:41 oak-gw06 kernel: LustreError: 2857:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88003d58e540) refcount = 2 Aug 8 07:29:41 oak-gw06 kernel: LustreError: 2857:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 07:29:41 oak-gw06 kernel: LustreError: 2857:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880041404e00/0xf077f1a82c860c48 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb86449de expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 07:29:41 oak-gw06 kernel: LustreError: 2857:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 07:34:49 oak-gw06 kernel: LustreError: 2869:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880186a1b480) refcount = 2 Aug 8 07:34:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 07:34:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 07:39:58 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 07:39:58 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 07:39:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502202898, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8804046be200/0xf077f1a82c860caa lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8653d33 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 07:39:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 07:39:58 oak-gw06 kernel: LustreError: 2873:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880186a1b9c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 07:39:58 oak-gw06 kernel: LustreError: 2873:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 07:39:58 oak-gw06 kernel: LustreError: 2873:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880186a1b9c0) refcount = 2 Aug 8 07:39:58 oak-gw06 kernel: LustreError: 2873:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 07:39:58 oak-gw06 kernel: LustreError: 2873:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8804046be200/0xf077f1a82c860caa lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8653d33 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 07:45:05 oak-gw06 kernel: LustreError: 2893:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880186a1b6c0) refcount = 2 Aug 8 07:45:05 oak-gw06 kernel: LustreError: 2893:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 07:45:05 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 07:45:05 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 07:50:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 07:50:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 07:50:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502203514, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880276f63800/0xf077f1a82c868b7c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8662e19 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 07:50:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 07:50:14 oak-gw06 kernel: LustreError: 2916:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88037aacacc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 07:50:14 oak-gw06 kernel: LustreError: 2916:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 07:50:14 oak-gw06 kernel: LustreError: 2916:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88037aacacc0) refcount = 2 Aug 8 07:50:14 oak-gw06 kernel: LustreError: 2916:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 07:50:14 oak-gw06 kernel: LustreError: 2916:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880276f63800/0xf077f1a82c868b7c lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8662e19 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 07:50:14 oak-gw06 kernel: LustreError: 2916:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 07:55:21 oak-gw06 kernel: LustreError: 2923:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88031a29e480) refcount = 2 Aug 8 07:55:21 oak-gw06 kernel: LustreError: 2923:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 07:55:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 07:55:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 08:00:27 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 08:00:27 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 08:00:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502204127, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800b2825c00/0xf077f1a82c87cea2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8671dee expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 08:00:27 oak-gw06 kernel: LustreError: 2935:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880235323600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 08:00:27 oak-gw06 kernel: LustreError: 2935:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 08:00:27 oak-gw06 kernel: LustreError: 2935:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880235323600) refcount = 2 Aug 8 08:00:27 oak-gw06 kernel: LustreError: 2935:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 08:00:27 oak-gw06 kernel: LustreError: 2935:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800b2825c00/0xf077f1a82c87cea2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8671dee expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 08:00:27 oak-gw06 kernel: LustreError: 2935:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 08:00:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 08:05:33 oak-gw06 kernel: LustreError: 2976:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803c62a0540) refcount = 2 Aug 8 08:05:33 oak-gw06 kernel: LustreError: 2976:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 08:05:33 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 08:05:33 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 08:10:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 08:10:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 08:10:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502204741, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880026e26800/0xf077f1a82c8835e2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8680ed4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 08:10:41 oak-gw06 kernel: LustreError: 2988:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802c56299c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 08:10:41 oak-gw06 kernel: LustreError: 2988:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 08:10:41 oak-gw06 kernel: LustreError: 2988:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802c56299c0) refcount = 2 Aug 8 08:10:41 oak-gw06 kernel: LustreError: 2988:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 08:10:41 oak-gw06 kernel: LustreError: 2988:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880026e26800/0xf077f1a82c8835e2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8680ed4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 08:10:41 oak-gw06 kernel: LustreError: 2988:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 08:10:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 08:15:51 oak-gw06 kernel: LustreError: 2993:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88037e1f2600) refcount = 2 Aug 8 08:15:51 oak-gw06 kernel: LustreError: 2993:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 08:15:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 08:15:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 08:21:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 08:21:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 08:21:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502205360, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880266143600/0xf077f1a82c887624 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8690309 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 08:21:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 08:21:00 oak-gw06 kernel: LustreError: 3008:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8804092b16c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 08:21:00 oak-gw06 kernel: LustreError: 3008:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 08:21:00 oak-gw06 kernel: LustreError: 3008:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8804092b16c0) refcount = 2 Aug 8 08:21:00 oak-gw06 kernel: LustreError: 3008:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 08:21:00 oak-gw06 kernel: LustreError: 3008:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880266143600/0xf077f1a82c887624 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8690309 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 08:21:00 oak-gw06 kernel: LustreError: 3008:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 08:26:06 oak-gw06 kernel: LustreError: 3012:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802836bf900) refcount = 2 Aug 8 08:26:06 oak-gw06 kernel: LustreError: 3012:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 08:26:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 08:26:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 08:31:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 08:31:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 08:31:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502205976, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88038362ce00/0xf077f1a82c88a318 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb869f435 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 08:31:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 08:31:16 oak-gw06 kernel: LustreError: 3025:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880354af09c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 08:31:16 oak-gw06 kernel: LustreError: 3025:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 08:31:16 oak-gw06 kernel: LustreError: 3025:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880354af09c0) refcount = 2 Aug 8 08:31:16 oak-gw06 kernel: LustreError: 3025:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 08:31:16 oak-gw06 kernel: LustreError: 3025:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88038362ce00/0xf077f1a82c88a318 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb869f435 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 08:31:16 oak-gw06 kernel: LustreError: 3025:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 08:36:23 oak-gw06 kernel: LustreError: 3033:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88018148cf00) refcount = 2 Aug 8 08:36:23 oak-gw06 kernel: LustreError: 3033:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 08:36:23 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 08:36:23 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 08:41:29 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 08:41:29 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 08:41:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502206589, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88038362ca00/0xf077f1a82c88ce6f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb86ae3fc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 08:41:29 oak-gw06 kernel: LustreError: 3045:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88018148cf00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 08:41:29 oak-gw06 kernel: LustreError: 3045:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 08:41:29 oak-gw06 kernel: LustreError: 3045:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88018148cf00) refcount = 2 Aug 8 08:41:29 oak-gw06 kernel: LustreError: 3045:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 08:41:29 oak-gw06 kernel: LustreError: 3045:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88038362ca00/0xf077f1a82c88ce6f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb86ae3fc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 08:41:29 oak-gw06 kernel: LustreError: 3045:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 08:41:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 08:46:38 oak-gw06 kernel: LustreError: 3050:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880345b243c0) refcount = 2 Aug 8 08:46:38 oak-gw06 kernel: LustreError: 3050:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 08:46:38 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 08:46:38 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 08:51:47 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 08:51:47 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 08:51:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502207207, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803f02ce400/0xf077f1a82c890859 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb86bd678 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 08:51:47 oak-gw06 kernel: LustreError: 3062:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802b0a38cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 08:51:47 oak-gw06 kernel: LustreError: 3062:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 08:51:47 oak-gw06 kernel: LustreError: 3062:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802b0a38cc0) refcount = 2 Aug 8 08:51:47 oak-gw06 kernel: LustreError: 3062:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 08:51:47 oak-gw06 kernel: LustreError: 3062:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803f02ce400/0xf077f1a82c890859 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb86bd678 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 08:51:47 oak-gw06 kernel: LustreError: 3062:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 08:51:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 08:56:55 oak-gw06 kernel: LustreError: 3069:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801a1f8a240) refcount = 2 Aug 8 08:56:55 oak-gw06 kernel: LustreError: 3069:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 08:56:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 08:56:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 09:02:01 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 09:02:01 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 09:02:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502207821, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801b127e800/0xf077f1a82c89362d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb86cc765 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 09:02:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 09:02:01 oak-gw06 kernel: LustreError: 3112:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803c62a0300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 09:02:01 oak-gw06 kernel: LustreError: 3112:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 09:02:01 oak-gw06 kernel: LustreError: 3112:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803c62a0300) refcount = 2 Aug 8 09:02:01 oak-gw06 kernel: LustreError: 3112:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 09:02:01 oak-gw06 kernel: LustreError: 3112:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801b127e800/0xf077f1a82c89362d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb86cc765 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 09:02:01 oak-gw06 kernel: LustreError: 3112:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 09:07:06 oak-gw06 kernel: LustreError: 3117:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880087d57e40) refcount = 2 Aug 8 09:07:06 oak-gw06 kernel: LustreError: 3117:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 09:07:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 09:07:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 09:12:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 09:12:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 09:12:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502208433, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802ef200c00/0xf077f1a82c896c97 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb86db795 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 09:12:13 oak-gw06 kernel: LustreError: 3131:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880284adf3c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 09:12:13 oak-gw06 kernel: LustreError: 3131:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 09:12:13 oak-gw06 kernel: LustreError: 3131:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880284adf3c0) refcount = 2 Aug 8 09:12:13 oak-gw06 kernel: LustreError: 3131:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 09:12:13 oak-gw06 kernel: LustreError: 3131:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802ef200c00/0xf077f1a82c896c97 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb86db795 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 09:12:13 oak-gw06 kernel: LustreError: 3131:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 09:12:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 09:17:19 oak-gw06 kernel: LustreError: 3135:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801e1ebeb40) refcount = 2 Aug 8 09:17:19 oak-gw06 kernel: LustreError: 3135:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 09:17:19 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 09:17:19 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 09:22:24 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 09:22:24 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 09:22:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502209044, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803c63fb400/0xf077f1a82c899d35 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb86ea5c6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 09:22:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 09:22:24 oak-gw06 kernel: LustreError: 3148:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801c5e3df00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 09:22:24 oak-gw06 kernel: LustreError: 3148:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 09:22:24 oak-gw06 kernel: LustreError: 3148:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c5e3df00) refcount = 2 Aug 8 09:22:24 oak-gw06 kernel: LustreError: 3148:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 09:22:24 oak-gw06 kernel: LustreError: 3148:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803c63fb400/0xf077f1a82c899d35 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb86ea5c6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 09:22:24 oak-gw06 kernel: LustreError: 3148:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 09:27:32 oak-gw06 kernel: LustreError: 3156:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88005a0f0780) refcount = 2 Aug 8 09:27:32 oak-gw06 kernel: LustreError: 3156:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 09:27:32 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 09:27:32 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 09:32:40 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 09:32:40 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 09:32:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502209660, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880221e48a00/0xf077f1a82c89d910 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb86f9803 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 09:32:40 oak-gw06 kernel: LustreError: 3168:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801b9e69480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 09:32:40 oak-gw06 kernel: LustreError: 3168:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 09:32:40 oak-gw06 kernel: LustreError: 3168:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801b9e69480) refcount = 2 Aug 8 09:32:40 oak-gw06 kernel: LustreError: 3168:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 09:32:40 oak-gw06 kernel: LustreError: 3168:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880221e48a00/0xf077f1a82c89d910 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb86f9803 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 09:32:40 oak-gw06 kernel: LustreError: 3168:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 09:32:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 09:37:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 09:37:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 09:42:56 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 09:42:56 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 09:42:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502210276, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88025c5ec400/0xf077f1a82c8ab749 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb87088cd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 09:42:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 09:42:56 oak-gw06 kernel: LustreError: 3197:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880203709480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 09:42:56 oak-gw06 kernel: LustreError: 3197:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880203709480) refcount = 2 Aug 8 09:42:56 oak-gw06 kernel: LustreError: 3197:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 09:42:56 oak-gw06 kernel: LustreError: 3197:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88025c5ec400/0xf077f1a82c8ab749 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb87088cd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 09:48:04 oak-gw06 kernel: LustreError: 3209:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88028bf13e40) refcount = 1 Aug 8 09:48:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 09:48:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 09:53:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 09:53:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 09:53:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502210891, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880195ad7800/0xf077f1a82c8bfd55 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8717855 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 09:53:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 09:53:11 oak-gw06 kernel: LustreError: 3228:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880065626840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 09:53:11 oak-gw06 kernel: LustreError: 3228:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 09:53:11 oak-gw06 kernel: LustreError: 3228:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880065626840) refcount = 2 Aug 8 09:53:11 oak-gw06 kernel: LustreError: 3228:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 09:53:11 oak-gw06 kernel: LustreError: 3228:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880195ad7800/0xf077f1a82c8bfd55 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8717855 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 09:58:21 oak-gw06 kernel: LustreError: 3233:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880252bb9cc0) refcount = 2 Aug 8 09:58:21 oak-gw06 kernel: LustreError: 3233:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 09:58:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 09:58:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 10:03:26 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 10:03:26 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 10:03:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502211506, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801b5886000/0xf077f1a82c8c5602 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb872691f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 10:03:26 oak-gw06 kernel: LustreError: 3281:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880252bb96c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 10:03:26 oak-gw06 kernel: LustreError: 3281:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 10:03:26 oak-gw06 kernel: LustreError: 3281:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880252bb96c0) refcount = 2 Aug 8 10:03:26 oak-gw06 kernel: LustreError: 3281:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 10:03:26 oak-gw06 kernel: LustreError: 3281:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801b5886000/0xf077f1a82c8c5602 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb872691f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 10:03:26 oak-gw06 kernel: LustreError: 3281:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 10:03:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 10:08:36 oak-gw06 kernel: LustreError: 3285:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880144938f00) refcount = 2 Aug 8 10:08:36 oak-gw06 kernel: LustreError: 3285:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 10:08:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 10:08:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 10:13:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 10:13:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 10:13:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502212122, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880327adcc00/0xf077f1a82c8cb22f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb87359c6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 10:13:42 oak-gw06 kernel: LustreError: 3296:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88015e3e5b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 10:13:42 oak-gw06 kernel: LustreError: 3296:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 10:13:42 oak-gw06 kernel: LustreError: 3296:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88015e3e5b40) refcount = 2 Aug 8 10:13:42 oak-gw06 kernel: LustreError: 3296:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 10:13:42 oak-gw06 kernel: LustreError: 3296:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880327adcc00/0xf077f1a82c8cb22f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb87359c6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 10:13:42 oak-gw06 kernel: LustreError: 3296:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 10:13:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 10:18:50 oak-gw06 kernel: LustreError: 3304:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803812dbc00) refcount = 2 Aug 8 10:18:50 oak-gw06 kernel: LustreError: 3304:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 10:18:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 10:18:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 10:23:55 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 10:23:55 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 10:23:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502212735, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88021d339e00/0xf077f1a82c8d0001 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8744852 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 10:23:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 10:23:55 oak-gw06 kernel: LustreError: 3315:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88035f26bd80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 10:23:55 oak-gw06 kernel: LustreError: 3315:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 10:23:55 oak-gw06 kernel: LustreError: 3315:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88035f26bd80) refcount = 2 Aug 8 10:23:55 oak-gw06 kernel: LustreError: 3315:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 10:23:55 oak-gw06 kernel: LustreError: 3315:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88021d339e00/0xf077f1a82c8d0001 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8744852 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 10:23:55 oak-gw06 kernel: LustreError: 3315:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 10:29:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 10:29:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 10:34:09 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 10:34:09 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 10:34:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502213349, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880154d7e800/0xf077f1a82c8d54b7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8753525 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 10:34:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 10:34:09 oak-gw06 kernel: LustreError: 3334:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880144938b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 10:34:09 oak-gw06 kernel: LustreError: 3334:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880144938b40) refcount = 2 Aug 8 10:34:09 oak-gw06 kernel: LustreError: 3334:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 10:34:09 oak-gw06 kernel: LustreError: 3334:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880154d7e800/0xf077f1a82c8d54b7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8753525 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 10:39:14 oak-gw06 kernel: LustreError: 3342:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880051f196c0) refcount = 2 Aug 8 10:39:14 oak-gw06 kernel: LustreError: 3342:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 10:39:14 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 10:39:14 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 10:44:23 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 10:44:23 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 10:44:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502213963, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880209f86000/0xf077f1a82c8dae28 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8762364 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 10:44:23 oak-gw06 kernel: LustreError: 3359:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8804186cb600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 10:44:23 oak-gw06 kernel: LustreError: 3359:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 10:44:23 oak-gw06 kernel: LustreError: 3359:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8804186cb600) refcount = 2 Aug 8 10:44:23 oak-gw06 kernel: LustreError: 3359:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 10:44:23 oak-gw06 kernel: LustreError: 3359:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880209f86000/0xf077f1a82c8dae28 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8762364 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 10:44:23 oak-gw06 kernel: LustreError: 3359:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 10:44:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 10:49:32 oak-gw06 kernel: LustreError: 3364:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801b63a53c0) refcount = 2 Aug 8 10:49:32 oak-gw06 kernel: LustreError: 3364:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 10:49:32 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 10:49:32 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 10:54:39 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 10:54:39 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 10:54:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502214579, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801e8ab6200/0xf077f1a82c8e1d10 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb87713a9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 10:54:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 10:54:39 oak-gw06 kernel: LustreError: 3378:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801969c3b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 10:54:39 oak-gw06 kernel: LustreError: 3378:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 10:54:39 oak-gw06 kernel: LustreError: 3378:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801969c3b40) refcount = 2 Aug 8 10:54:39 oak-gw06 kernel: LustreError: 3378:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 10:54:39 oak-gw06 kernel: LustreError: 3378:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801e8ab6200/0xf077f1a82c8e1d10 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb87713a9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 10:54:39 oak-gw06 kernel: LustreError: 3378:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 10:59:48 oak-gw06 kernel: LustreError: 3382:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88029468a540) refcount = 2 Aug 8 10:59:48 oak-gw06 kernel: LustreError: 3382:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 10:59:48 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 10:59:48 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 11:04:54 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 11:04:54 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 11:04:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502215194, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802f3a37000/0xf077f1a82c8e7e53 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8780369 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 11:04:54 oak-gw06 kernel: LustreError: 3429:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88038cec2600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 11:04:54 oak-gw06 kernel: LustreError: 3429:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 11:04:54 oak-gw06 kernel: LustreError: 3429:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88038cec2600) refcount = 2 Aug 8 11:04:54 oak-gw06 kernel: LustreError: 3429:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 11:04:54 oak-gw06 kernel: LustreError: 3429:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802f3a37000/0xf077f1a82c8e7e53 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8780369 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 11:04:54 oak-gw06 kernel: LustreError: 3429:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 11:04:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 11:10:01 oak-gw06 kernel: LustreError: 3436:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801f3249840) refcount = 2 Aug 8 11:10:01 oak-gw06 kernel: LustreError: 3436:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 11:10:01 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 11:10:01 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 11:15:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 11:15:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 11:15:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502215811, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801b9f7fa00/0xf077f1a82c8edaf7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb878f30d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 11:15:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 11:15:11 oak-gw06 kernel: LustreError: 3446:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88028ea5bc00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 11:15:11 oak-gw06 kernel: LustreError: 3446:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 11:15:11 oak-gw06 kernel: LustreError: 3446:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88028ea5bc00) refcount = 2 Aug 8 11:15:11 oak-gw06 kernel: LustreError: 3446:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 11:15:11 oak-gw06 kernel: LustreError: 3446:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801b9f7fa00/0xf077f1a82c8edaf7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb878f30d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 11:15:11 oak-gw06 kernel: LustreError: 3446:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 11:20:18 oak-gw06 kernel: LustreError: 3457:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880046a2dcc0) refcount = 2 Aug 8 11:20:18 oak-gw06 kernel: LustreError: 3457:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 11:20:18 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 11:20:18 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 11:25:25 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 11:25:25 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 11:25:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502216425, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802ef201800/0xf077f1a82c8f11d8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb879e279 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 11:25:25 oak-gw06 kernel: LustreError: 3463:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803c0f8a000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 11:25:25 oak-gw06 kernel: LustreError: 3463:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 11:25:25 oak-gw06 kernel: LustreError: 3463:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803c0f8a000) refcount = 2 Aug 8 11:25:25 oak-gw06 kernel: LustreError: 3463:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 11:25:25 oak-gw06 kernel: LustreError: 3463:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802ef201800/0xf077f1a82c8f11d8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb879e279 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 11:25:25 oak-gw06 kernel: LustreError: 3463:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 11:25:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 11:30:33 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 11:30:33 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 11:35:43 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 11:35:43 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 11:35:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502217043, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802a83b8600/0xf077f1a82c8f9128 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb87ad3ba expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 11:35:43 oak-gw06 kernel: LustreError: 3483:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880365bf63c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 11:35:43 oak-gw06 kernel: LustreError: 3483:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880365bf63c0) refcount = 2 Aug 8 11:35:43 oak-gw06 kernel: LustreError: 3483:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 11:35:43 oak-gw06 kernel: LustreError: 3483:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802a83b8600/0xf077f1a82c8f9128 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb87ad3ba expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 11:35:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 11:40:53 oak-gw06 kernel: LustreError: 3495:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880365bf6b40) refcount = 2 Aug 8 11:40:53 oak-gw06 kernel: LustreError: 3495:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 11:40:53 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 11:40:53 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 11:46:01 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 11:46:01 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 11:46:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502217661, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803af0b6200/0xf077f1a82c8fa33b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb87bc3f1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 11:46:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 11:46:01 oak-gw06 kernel: LustreError: 3499:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801f77e6f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 11:46:01 oak-gw06 kernel: LustreError: 3499:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 11:46:01 oak-gw06 kernel: LustreError: 3499:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801f77e6f00) refcount = 2 Aug 8 11:46:01 oak-gw06 kernel: LustreError: 3499:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 11:46:01 oak-gw06 kernel: LustreError: 3499:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803af0b6200/0xf077f1a82c8fa33b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb87bc3f1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 11:46:01 oak-gw06 kernel: LustreError: 3499:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 11:51:10 oak-gw06 kernel: LustreError: 3510:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880168a00900) refcount = 1 Aug 8 11:51:10 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 11:51:10 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 11:56:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 11:56:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 11:56:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502218276, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802078d0200/0xf077f1a82c8fcdf1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb87cb309 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 11:56:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 11:56:16 oak-gw06 kernel: LustreError: 3517:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802a5bca9c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 11:56:16 oak-gw06 kernel: LustreError: 3517:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 11:56:16 oak-gw06 kernel: LustreError: 3517:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a5bca9c0) refcount = 2 Aug 8 11:56:16 oak-gw06 kernel: LustreError: 3517:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 11:56:16 oak-gw06 kernel: LustreError: 3517:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802078d0200/0xf077f1a82c8fcdf1 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb87cb309 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 12:01:23 oak-gw06 kernel: LustreError: 3563:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88027eed1f00) refcount = 2 Aug 8 12:01:23 oak-gw06 kernel: LustreError: 3563:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 12:01:23 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 12:01:23 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 12:06:30 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 12:06:30 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 12:06:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502218890, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802f3a37400/0xf077f1a82c8ffd46 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb87da1a3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 12:06:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 12:06:30 oak-gw06 kernel: LustreError: 3569:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88027eed1180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 12:06:30 oak-gw06 kernel: LustreError: 3569:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 12:06:30 oak-gw06 kernel: LustreError: 3569:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88027eed1180) refcount = 2 Aug 8 12:06:30 oak-gw06 kernel: LustreError: 3569:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 12:06:30 oak-gw06 kernel: LustreError: 3569:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802f3a37400/0xf077f1a82c8ffd46 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb87da1a3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 12:06:30 oak-gw06 kernel: LustreError: 3569:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 12:11:37 oak-gw06 kernel: LustreError: 3586:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff88013f849840) refcount = 2 Aug 8 12:11:37 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 12:11:37 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 12:16:45 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 12:16:45 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 12:16:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502219505, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802873d6000/0xf077f1a82c9053ed lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb87e9163 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 12:16:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 12:16:45 oak-gw06 kernel: LustreError: 3590:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88013f849000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 12:16:45 oak-gw06 kernel: LustreError: 3590:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 12:16:45 oak-gw06 kernel: LustreError: 3590:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88013f849000) refcount = 2 Aug 8 12:16:45 oak-gw06 kernel: LustreError: 3590:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 12:16:45 oak-gw06 kernel: LustreError: 3590:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802873d6000/0xf077f1a82c9053ed lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb87e9163 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 12:21:54 oak-gw06 kernel: LustreError: 3604:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803097bbd80) refcount = 2 Aug 8 12:21:54 oak-gw06 kernel: LustreError: 3604:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 12:21:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 12:21:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 12:27:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 12:27:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 12:27:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502220120, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803c63f9000/0xf077f1a82c90a3d3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb87f8027 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 12:27:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 12:27:00 oak-gw06 kernel: LustreError: 3608:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803b3e5df00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 12:27:00 oak-gw06 kernel: LustreError: 3608:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 12:27:00 oak-gw06 kernel: LustreError: 3608:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803b3e5df00) refcount = 2 Aug 8 12:27:00 oak-gw06 kernel: LustreError: 3608:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 12:27:00 oak-gw06 kernel: LustreError: 3608:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803c63f9000/0xf077f1a82c90a3d3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb87f8027 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 12:27:00 oak-gw06 kernel: LustreError: 3608:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 12:32:08 oak-gw06 kernel: LustreError: 3627:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880054184000) refcount = 2 Aug 8 12:32:08 oak-gw06 kernel: LustreError: 3627:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 12:32:08 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 12:32:08 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 12:37:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 12:37:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 12:37:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502220734, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88028fa7ac00/0xf077f1a82c90f739 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8806eb3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 12:37:14 oak-gw06 kernel: LustreError: 3631:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801365379c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 12:37:14 oak-gw06 kernel: LustreError: 3631:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 12:37:14 oak-gw06 kernel: LustreError: 3631:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801365379c0) refcount = 2 Aug 8 12:37:14 oak-gw06 kernel: LustreError: 3631:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 12:37:14 oak-gw06 kernel: LustreError: 3631:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88028fa7ac00/0xf077f1a82c90f739 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8806eb3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 12:37:14 oak-gw06 kernel: LustreError: 3631:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 12:37:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 12:42:20 oak-gw06 kernel: LustreError: 3644:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802866ba600) refcount = 2 Aug 8 12:42:20 oak-gw06 kernel: LustreError: 3644:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 12:42:20 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 12:42:20 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 12:47:26 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 12:47:26 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 12:47:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502221346, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802ee628c00/0xf077f1a82c912e2f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8815ccf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 12:47:26 oak-gw06 kernel: LustreError: 3652:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880004639240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 12:47:26 oak-gw06 kernel: LustreError: 3652:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 12:47:26 oak-gw06 kernel: LustreError: 3652:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880004639240) refcount = 2 Aug 8 12:47:26 oak-gw06 kernel: LustreError: 3652:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 12:47:26 oak-gw06 kernel: LustreError: 3652:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802ee628c00/0xf077f1a82c912e2f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8815ccf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 12:47:26 oak-gw06 kernel: LustreError: 3652:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 12:47:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 12:52:33 oak-gw06 kernel: LustreError: 3663:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007c9fa900) refcount = 2 Aug 8 12:52:33 oak-gw06 kernel: LustreError: 3663:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 12:52:33 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 12:52:33 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 12:57:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 12:57:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 12:57:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502221962, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803d3b93000/0xf077f1a82c917643 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8824d5a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 12:57:42 oak-gw06 kernel: LustreError: 3671:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88041d1e0900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 12:57:42 oak-gw06 kernel: LustreError: 3671:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 12:57:42 oak-gw06 kernel: LustreError: 3671:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041d1e0900) refcount = 2 Aug 8 12:57:42 oak-gw06 kernel: LustreError: 3671:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 12:57:42 oak-gw06 kernel: LustreError: 3671:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803d3b93000/0xf077f1a82c917643 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8824d5a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 12:57:42 oak-gw06 kernel: LustreError: 3671:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 12:57:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 13:02:51 oak-gw06 kernel: LustreError: 3713:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a5bca240) refcount = 2 Aug 8 13:02:51 oak-gw06 kernel: LustreError: 3713:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 13:02:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 13:02:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 13:07:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 13:07:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 13:07:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502222579, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802fffca000/0xf077f1a82c91b597 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8833fcf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 13:07:59 oak-gw06 kernel: LustreError: 3717:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88013f472540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 13:07:59 oak-gw06 kernel: LustreError: 3717:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 13:07:59 oak-gw06 kernel: LustreError: 3717:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88013f472540) refcount = 2 Aug 8 13:07:59 oak-gw06 kernel: LustreError: 3717:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 13:07:59 oak-gw06 kernel: LustreError: 3717:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802fffca000/0xf077f1a82c91b597 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8833fcf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 13:07:59 oak-gw06 kernel: LustreError: 3717:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 13:07:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 13:13:09 oak-gw06 kernel: LustreError: 3741:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880195348240) refcount = 2 Aug 8 13:13:09 oak-gw06 kernel: LustreError: 3741:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 13:13:09 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 13:13:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 13:18:19 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 13:18:19 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 13:18:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502223199, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803c0ed1600/0xf077f1a82c92bd83 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb88432c2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 13:18:19 oak-gw06 kernel: LustreError: 3745:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88040033cb40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 13:18:19 oak-gw06 kernel: LustreError: 3745:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 13:18:19 oak-gw06 kernel: LustreError: 3745:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88040033cb40) refcount = 2 Aug 8 13:18:19 oak-gw06 kernel: LustreError: 3745:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 13:18:19 oak-gw06 kernel: LustreError: 3745:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803c0ed1600/0xf077f1a82c92bd83 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb88432c2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 13:18:19 oak-gw06 kernel: LustreError: 3745:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 13:18:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 13:23:24 oak-gw06 kernel: LustreError: 3762:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803097bba80) refcount = 2 Aug 8 13:23:24 oak-gw06 kernel: LustreError: 3762:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 13:23:24 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 13:23:24 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 13:28:32 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 13:28:32 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 13:28:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502223812, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88041de50e00/0xf077f1a82c92fabc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb88520f3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 13:28:32 oak-gw06 kernel: LustreError: 3767:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802937220c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 13:28:32 oak-gw06 kernel: LustreError: 3767:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 13:28:32 oak-gw06 kernel: LustreError: 3767:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802937220c0) refcount = 2 Aug 8 13:28:32 oak-gw06 kernel: LustreError: 3767:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 13:28:32 oak-gw06 kernel: LustreError: 3767:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88041de50e00/0xf077f1a82c92fabc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb88520f3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 13:28:32 oak-gw06 kernel: LustreError: 3767:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 13:28:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 13:33:40 oak-gw06 kernel: LustreError: 3782:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880013056a80) refcount = 2 Aug 8 13:33:40 oak-gw06 kernel: LustreError: 3782:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 13:33:40 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 13:33:40 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 13:38:48 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 13:38:48 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 13:38:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502224428, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880209f85e00/0xf077f1a82c934ce0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8861226 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 13:38:48 oak-gw06 kernel: LustreError: 3787:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801b7b1c3c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 13:38:48 oak-gw06 kernel: LustreError: 3787:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 13:38:48 oak-gw06 kernel: LustreError: 3787:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801b7b1c3c0) refcount = 2 Aug 8 13:38:48 oak-gw06 kernel: LustreError: 3787:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 13:38:48 oak-gw06 kernel: LustreError: 3787:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880209f85e00/0xf077f1a82c934ce0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8861226 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 13:38:48 oak-gw06 kernel: LustreError: 3787:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 13:38:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 13:43:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 13:43:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 13:49:04 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 13:49:04 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 13:49:04 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502225044, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880285663a00/0xf077f1a82c9381c9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb88702e2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 13:49:04 oak-gw06 kernel: LustreError: 3807:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803de20bc00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 13:49:04 oak-gw06 kernel: LustreError: 3807:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803de20bc00) refcount = 2 Aug 8 13:49:04 oak-gw06 kernel: LustreError: 3807:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 13:49:04 oak-gw06 kernel: LustreError: 3807:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880285663a00/0xf077f1a82c9381c9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb88702e2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 13:49:04 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 13:54:14 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 13:54:14 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 13:59:23 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 13:59:23 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 13:59:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502225663, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88027a379200/0xf077f1a82c93a818 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb887f6df expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 13:59:23 oak-gw06 kernel: LustreError: 3827:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880285b84540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 13:59:23 oak-gw06 kernel: LustreError: 3827:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880285b84540) refcount = 2 Aug 8 13:59:23 oak-gw06 kernel: LustreError: 3827:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 13:59:23 oak-gw06 kernel: LustreError: 3827:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88027a379200/0xf077f1a82c93a818 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb887f6df expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 13:59:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 14:04:30 oak-gw06 kernel: LustreError: 3872:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041932e3c0) refcount = 2 Aug 8 14:04:30 oak-gw06 kernel: LustreError: 3872:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 14:04:30 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 14:04:30 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 14:09:38 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 14:09:38 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 14:09:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502226278, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801ce2a7600/0xf077f1a82c942f17 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb888e84a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 14:09:38 oak-gw06 kernel: LustreError: 3880:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880413221e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 14:09:38 oak-gw06 kernel: LustreError: 3880:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 14:09:38 oak-gw06 kernel: LustreError: 3880:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413221e40) refcount = 2 Aug 8 14:09:38 oak-gw06 kernel: LustreError: 3880:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 14:09:38 oak-gw06 kernel: LustreError: 3880:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801ce2a7600/0xf077f1a82c942f17 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb888e84a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 14:09:38 oak-gw06 kernel: LustreError: 3880:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 14:09:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 14:14:45 oak-gw06 kernel: LustreError: 3891:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022a2d4e40) refcount = 2 Aug 8 14:14:45 oak-gw06 kernel: LustreError: 3891:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 14:14:45 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 14:14:45 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 14:19:51 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 14:19:51 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 14:19:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502226891, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880240ef5200/0xf077f1a82c94610c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb889d8e3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 14:19:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 14:19:51 oak-gw06 kernel: LustreError: 3906:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88013beed000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 14:19:51 oak-gw06 kernel: LustreError: 3906:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 14:19:51 oak-gw06 kernel: LustreError: 3906:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88013beed000) refcount = 2 Aug 8 14:19:51 oak-gw06 kernel: LustreError: 3906:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 14:19:51 oak-gw06 kernel: LustreError: 3906:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880240ef5200/0xf077f1a82c94610c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb889d8e3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 14:19:51 oak-gw06 kernel: LustreError: 3906:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 14:24:56 oak-gw06 kernel: LustreError: 3918:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880127eff240) refcount = 2 Aug 8 14:24:56 oak-gw06 kernel: LustreError: 3918:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 14:24:56 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 14:24:56 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 14:30:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 14:30:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 14:30:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502227502, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801f0f44c00/0xf077f1a82c94c78f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb88ac810 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 14:30:02 oak-gw06 kernel: LustreError: 3934:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880168b98540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 14:30:02 oak-gw06 kernel: LustreError: 3934:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 14:30:02 oak-gw06 kernel: LustreError: 3934:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880168b98540) refcount = 2 Aug 8 14:30:02 oak-gw06 kernel: LustreError: 3934:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 14:30:02 oak-gw06 kernel: LustreError: 3934:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801f0f44c00/0xf077f1a82c94c78f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb88ac810 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 14:30:02 oak-gw06 kernel: LustreError: 3934:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 14:30:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 14:35:07 oak-gw06 kernel: LustreError: 3937:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880407f1d000) refcount = 2 Aug 8 14:35:07 oak-gw06 kernel: LustreError: 3937:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 14:35:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 14:35:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 14:40:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 14:40:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 14:40:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502228114, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880143525a00/0xf077f1a82c954056 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb88bb990 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 14:40:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 14:40:14 oak-gw06 kernel: LustreError: 3952:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880407f1d300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 14:40:14 oak-gw06 kernel: LustreError: 3952:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 14:40:14 oak-gw06 kernel: LustreError: 3952:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880407f1d300) refcount = 2 Aug 8 14:40:14 oak-gw06 kernel: LustreError: 3952:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 14:40:14 oak-gw06 kernel: LustreError: 3952:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880143525a00/0xf077f1a82c954056 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb88bb990 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 14:40:14 oak-gw06 kernel: LustreError: 3952:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 14:45:23 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 14:45:23 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 14:50:30 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 14:50:30 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 14:50:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502228730, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880327adec00/0xf077f1a82c95855a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb88cac44 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 14:50:30 oak-gw06 kernel: LustreError: 3972:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88042c239d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 14:50:30 oak-gw06 kernel: LustreError: 3972:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042c239d80) refcount = 2 Aug 8 14:50:30 oak-gw06 kernel: LustreError: 3972:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 14:50:30 oak-gw06 kernel: LustreError: 3972:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880327adec00/0xf077f1a82c95855a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb88cac44 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 14:50:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 14:55:37 oak-gw06 kernel: LustreError: 3975:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041728c780) refcount = 2 Aug 8 14:55:37 oak-gw06 kernel: LustreError: 3975:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 14:55:37 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 14:55:37 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 15:00:43 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 15:00:43 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 15:00:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502229343, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880282225000/0xf077f1a82c95bc6c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb88d9eab expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 15:00:43 oak-gw06 kernel: LustreError: 3989:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801b4699b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 15:00:43 oak-gw06 kernel: LustreError: 3989:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 15:00:43 oak-gw06 kernel: LustreError: 3989:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801b4699b40) refcount = 2 Aug 8 15:00:43 oak-gw06 kernel: LustreError: 3989:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 15:00:43 oak-gw06 kernel: LustreError: 3989:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880282225000/0xf077f1a82c95bc6c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb88d9eab expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 15:00:43 oak-gw06 kernel: LustreError: 3989:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 15:00:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 15:05:52 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 15:05:52 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 15:10:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 15:10:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 15:10:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502229957, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802dc985c00/0xf077f1a82c960051 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb88e8ff3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 15:10:57 oak-gw06 kernel: LustreError: 4040:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88024a1909c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 15:10:57 oak-gw06 kernel: LustreError: 4040:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88024a1909c0) refcount = 2 Aug 8 15:10:57 oak-gw06 kernel: LustreError: 4040:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 15:10:57 oak-gw06 kernel: LustreError: 4040:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802dc985c00/0xf077f1a82c960051 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb88e8ff3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 15:10:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 15:16:06 oak-gw06 kernel: LustreError: 4044:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042c2396c0) refcount = 2 Aug 8 15:16:06 oak-gw06 kernel: LustreError: 4044:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 15:16:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 15:16:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 15:21:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 15:21:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 15:21:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502230573, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880359498a00/0xf077f1a82c9647a8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb88f8309 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 15:21:13 oak-gw06 kernel: LustreError: 4060:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88014c884c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 15:21:13 oak-gw06 kernel: LustreError: 4060:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 15:21:13 oak-gw06 kernel: LustreError: 4060:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88014c884c00) refcount = 2 Aug 8 15:21:13 oak-gw06 kernel: LustreError: 4060:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 15:21:13 oak-gw06 kernel: LustreError: 4060:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880359498a00/0xf077f1a82c9647a8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb88f8309 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 15:21:13 oak-gw06 kernel: LustreError: 4060:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 15:21:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 15:26:22 oak-gw06 kernel: LustreError: 4063:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800656263c0) refcount = 2 Aug 8 15:26:22 oak-gw06 kernel: LustreError: 4063:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 15:26:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 15:26:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 15:31:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 15:31:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 15:31:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502231188, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803fc416400/0xf077f1a82c9679f1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb890771b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 15:31:28 oak-gw06 kernel: LustreError: 4075:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88004f06dd80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 15:31:28 oak-gw06 kernel: LustreError: 4075:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 15:31:28 oak-gw06 kernel: LustreError: 4075:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88004f06dd80) refcount = 2 Aug 8 15:31:28 oak-gw06 kernel: LustreError: 4075:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 15:31:28 oak-gw06 kernel: LustreError: 4075:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803fc416400/0xf077f1a82c9679f1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb890771b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 15:31:28 oak-gw06 kernel: LustreError: 4075:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 15:31:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 15:36:35 oak-gw06 kernel: LustreError: 4092:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e46d6cc0) refcount = 2 Aug 8 15:36:35 oak-gw06 kernel: LustreError: 4092:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 15:36:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 15:36:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 15:41:43 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 15:41:43 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 15:41:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502231803, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801d5366000/0xf077f1a82c979310 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8916bab expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 15:41:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 15:41:43 oak-gw06 kernel: LustreError: 4106:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802a707b480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 15:41:43 oak-gw06 kernel: LustreError: 4106:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 15:41:43 oak-gw06 kernel: LustreError: 4106:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a707b480) refcount = 2 Aug 8 15:41:43 oak-gw06 kernel: LustreError: 4106:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 15:41:43 oak-gw06 kernel: LustreError: 4106:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801d5366000/0xf077f1a82c979310 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8916bab expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 15:41:43 oak-gw06 kernel: LustreError: 4106:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 15:46:49 oak-gw06 kernel: LustreError: 4111:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802717e2f00) refcount = 2 Aug 8 15:46:49 oak-gw06 kernel: LustreError: 4111:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 15:46:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 15:46:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 15:51:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 15:51:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 15:51:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502232417, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802ef201a00/0xf077f1a82c97e19f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8925e97 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 15:51:57 oak-gw06 kernel: LustreError: 4121:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801e6f64600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 15:51:57 oak-gw06 kernel: LustreError: 4121:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 15:51:57 oak-gw06 kernel: LustreError: 4121:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801e6f64600) refcount = 2 Aug 8 15:51:57 oak-gw06 kernel: LustreError: 4121:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 15:51:57 oak-gw06 kernel: LustreError: 4121:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802ef201a00/0xf077f1a82c97e19f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8925e97 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 15:51:57 oak-gw06 kernel: LustreError: 4121:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 15:51:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 15:57:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 15:57:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 16:02:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 16:02:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 16:02:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502233033, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88028a2a1600/0xf077f1a82c98126e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb89352da expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 16:02:13 oak-gw06 kernel: LustreError: 4175:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801f96f4c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 16:02:13 oak-gw06 kernel: LustreError: 4175:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801f96f4c00) refcount = 2 Aug 8 16:02:13 oak-gw06 kernel: LustreError: 4175:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 16:02:13 oak-gw06 kernel: LustreError: 4175:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88028a2a1600/0xf077f1a82c98126e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb89352da expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 16:02:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 16:07:21 oak-gw06 kernel: LustreError: 4198:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88019365ad80) refcount = 2 Aug 8 16:07:21 oak-gw06 kernel: LustreError: 4198:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 16:07:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 16:07:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 16:12:27 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 16:12:27 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 16:12:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502233647, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801569e1400/0xf077f1a82c988a63 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb89446c2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 16:12:27 oak-gw06 kernel: LustreError: 4214:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88031a29e0c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 16:12:27 oak-gw06 kernel: LustreError: 4214:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 16:12:27 oak-gw06 kernel: LustreError: 4214:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88031a29e0c0) refcount = 2 Aug 8 16:12:27 oak-gw06 kernel: LustreError: 4214:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 16:12:27 oak-gw06 kernel: LustreError: 4214:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801569e1400/0xf077f1a82c988a63 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb89446c2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 16:12:27 oak-gw06 kernel: LustreError: 4214:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 16:12:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 16:17:37 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 16:17:37 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 16:22:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 16:22:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 16:22:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502234264, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880292ac3400/0xf077f1a82c98ed04 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8953b44 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 16:22:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 16:22:44 oak-gw06 kernel: LustreError: 4234:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802d1b8f6c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 16:22:44 oak-gw06 kernel: LustreError: 4234:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802d1b8f6c0) refcount = 2 Aug 8 16:22:44 oak-gw06 kernel: LustreError: 4234:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 16:22:44 oak-gw06 kernel: LustreError: 4234:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880292ac3400/0xf077f1a82c98ed04 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8953b44 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 16:27:50 oak-gw06 kernel: LustreError: 4243:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803adc8e480) refcount = 2 Aug 8 16:27:50 oak-gw06 kernel: LustreError: 4243:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 16:27:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 16:27:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 16:32:56 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 16:32:56 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 16:32:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502234876, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880402fd3a00/0xf077f1a82c995d20 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8962da4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 16:32:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 16:32:56 oak-gw06 kernel: LustreError: 4268:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802866efcc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 16:32:56 oak-gw06 kernel: LustreError: 4268:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 16:32:56 oak-gw06 kernel: LustreError: 4268:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802866efcc0) refcount = 2 Aug 8 16:32:56 oak-gw06 kernel: LustreError: 4268:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 16:32:56 oak-gw06 kernel: LustreError: 4268:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880402fd3a00/0xf077f1a82c995d20 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8962da4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 16:32:56 oak-gw06 kernel: LustreError: 4268:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 16:38:05 oak-gw06 kernel: LustreError: 4270:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802957110c0) refcount = 2 Aug 8 16:38:05 oak-gw06 kernel: LustreError: 4270:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 16:38:05 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 16:38:05 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 16:43:10 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 16:43:10 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 16:43:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502235490, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88029330be00/0xf077f1a82c998720 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb897211c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 16:43:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 16:43:10 oak-gw06 kernel: LustreError: 4282:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801220e8f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 16:43:10 oak-gw06 kernel: LustreError: 4282:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 16:43:10 oak-gw06 kernel: LustreError: 4282:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801220e8f00) refcount = 2 Aug 8 16:43:10 oak-gw06 kernel: LustreError: 4282:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 16:43:10 oak-gw06 kernel: LustreError: 4282:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88029330be00/0xf077f1a82c998720 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb897211c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 16:43:10 oak-gw06 kernel: LustreError: 4282:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 16:48:18 oak-gw06 kernel: LustreError: 4293:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880185c81a80) refcount = 2 Aug 8 16:48:18 oak-gw06 kernel: LustreError: 4293:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 16:48:18 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 16:48:18 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 16:53:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 16:53:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 16:53:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502236108, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88014c659400/0xf077f1a82c99c7b6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb898163f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 16:53:28 oak-gw06 kernel: LustreError: 4308:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880412ef5f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 16:53:28 oak-gw06 kernel: LustreError: 4308:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 16:53:28 oak-gw06 kernel: LustreError: 4308:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880412ef5f00) refcount = 2 Aug 8 16:53:28 oak-gw06 kernel: LustreError: 4308:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 16:53:28 oak-gw06 kernel: LustreError: 4308:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88014c659400/0xf077f1a82c99c7b6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb898163f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 16:53:28 oak-gw06 kernel: LustreError: 4308:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 16:53:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 16:58:34 oak-gw06 kernel: LustreError: 4313:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88035ff4f180) refcount = 2 Aug 8 16:58:34 oak-gw06 kernel: LustreError: 4313:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 16:58:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 16:58:34 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 17:03:40 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 17:03:40 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 17:03:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502236720, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880111e31c00/0xf077f1a82c99fd24 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb89909fd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 17:03:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 17:03:40 oak-gw06 kernel: LustreError: 4356:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880361f850c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 17:03:40 oak-gw06 kernel: LustreError: 4356:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 17:03:40 oak-gw06 kernel: LustreError: 4356:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880361f850c0) refcount = 2 Aug 8 17:03:40 oak-gw06 kernel: LustreError: 4356:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 17:03:40 oak-gw06 kernel: LustreError: 4356:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880111e31c00/0xf077f1a82c99fd24 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb89909fd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 17:03:40 oak-gw06 kernel: LustreError: 4356:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 17:08:46 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 17:08:46 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 17:13:53 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 17:13:53 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 17:13:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502237333, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802bdfc7600/0xf077f1a82c9a334f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb899fd3d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 17:13:53 oak-gw06 kernel: LustreError: 4371:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88042b68ff00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 17:13:53 oak-gw06 kernel: LustreError: 4371:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042b68ff00) refcount = 2 Aug 8 17:13:53 oak-gw06 kernel: LustreError: 4371:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 17:13:53 oak-gw06 kernel: LustreError: 4371:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802bdfc7600/0xf077f1a82c9a334f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb899fd3d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 17:13:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 17:19:01 oak-gw06 kernel: LustreError: 4379:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880204bccc00) refcount = 2 Aug 8 17:19:01 oak-gw06 kernel: LustreError: 4379:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 17:19:01 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 17:19:01 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 17:24:08 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 17:24:08 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 17:24:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502237948, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88034b250000/0xf077f1a82c9a7670 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb89af20c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 17:24:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 17:24:08 oak-gw06 kernel: LustreError: 4389:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88027d89af00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 17:24:08 oak-gw06 kernel: LustreError: 4389:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 17:24:08 oak-gw06 kernel: LustreError: 4389:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88027d89af00) refcount = 2 Aug 8 17:24:08 oak-gw06 kernel: LustreError: 4389:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 17:24:08 oak-gw06 kernel: LustreError: 4389:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88034b250000/0xf077f1a82c9a7670 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb89af20c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 17:24:08 oak-gw06 kernel: LustreError: 4389:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 17:29:14 oak-gw06 kernel: LustreError: 4393:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c5a38540) refcount = 2 Aug 8 17:29:14 oak-gw06 kernel: LustreError: 4393:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 17:29:14 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 17:29:14 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 17:34:22 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 17:34:22 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 17:34:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502238562, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880072f26200/0xf077f1a82c9a9143 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb89be6e9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 17:34:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 17:34:22 oak-gw06 kernel: LustreError: 4404:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801c5a38000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 17:34:22 oak-gw06 kernel: LustreError: 4404:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 17:34:22 oak-gw06 kernel: LustreError: 4404:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c5a38000) refcount = 2 Aug 8 17:34:22 oak-gw06 kernel: LustreError: 4404:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 17:34:22 oak-gw06 kernel: LustreError: 4404:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880072f26200/0xf077f1a82c9a9143 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb89be6e9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 17:34:22 oak-gw06 kernel: LustreError: 4404:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 17:39:32 oak-gw06 kernel: LustreError: 4411:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801a2c916c0) refcount = 2 Aug 8 17:39:32 oak-gw06 kernel: LustreError: 4411:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 17:39:32 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 17:39:32 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 17:44:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 17:44:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 17:44:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502239181, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802a32b0200/0xf077f1a82c9accf4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb89cdc44 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 17:44:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 17:44:41 oak-gw06 kernel: LustreError: 4424:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88019780ac00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 17:44:41 oak-gw06 kernel: LustreError: 4424:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 17:44:41 oak-gw06 kernel: LustreError: 4424:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88019780ac00) refcount = 2 Aug 8 17:44:41 oak-gw06 kernel: LustreError: 4424:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 17:44:41 oak-gw06 kernel: LustreError: 4424:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802a32b0200/0xf077f1a82c9accf4 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb89cdc44 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 17:44:41 oak-gw06 kernel: LustreError: 4424:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 17:49:48 oak-gw06 kernel: LustreError: 4432:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801fb8886c0) refcount = 2 Aug 8 17:49:48 oak-gw06 kernel: LustreError: 4432:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 17:49:48 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 17:49:48 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 17:54:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 17:54:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 17:54:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502239797, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801a66f2a00/0xf077f1a82c9b138e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb89dd1ec expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 17:54:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 17:54:57 oak-gw06 kernel: LustreError: 4442:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88041d5d49c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 17:54:57 oak-gw06 kernel: LustreError: 4442:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 17:54:57 oak-gw06 kernel: LustreError: 4442:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041d5d49c0) refcount = 2 Aug 8 17:54:57 oak-gw06 kernel: LustreError: 4442:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 17:54:57 oak-gw06 kernel: LustreError: 4442:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801a66f2a00/0xf077f1a82c9b138e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb89dd1ec expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 17:54:57 oak-gw06 kernel: LustreError: 4442:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 18:00:07 oak-gw06 kernel: LustreError: 4453:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880168a603c0) refcount = 1 Aug 8 18:00:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 18:00:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 18:05:15 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 18:05:15 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 18:05:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502240415, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880315a60000/0xf077f1a82c9b4505 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb89ec786 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 18:05:15 oak-gw06 kernel: LustreError: 4490:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ad277d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 18:05:15 oak-gw06 kernel: LustreError: 4490:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 18:05:15 oak-gw06 kernel: LustreError: 4490:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ad277d80) refcount = 2 Aug 8 18:05:15 oak-gw06 kernel: LustreError: 4490:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 18:05:15 oak-gw06 kernel: LustreError: 4490:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880315a60000/0xf077f1a82c9b4505 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb89ec786 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 18:05:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 18:10:22 oak-gw06 kernel: LustreError: 4505:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ac504600) refcount = 2 Aug 8 18:10:22 oak-gw06 kernel: LustreError: 4505:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 18:10:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 18:10:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 18:15:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 18:15:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 18:15:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502241028, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880111e33c00/0xf077f1a82c9b855c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb89fbb28 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 18:15:28 oak-gw06 kernel: LustreError: 4509:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802164f6180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 18:15:28 oak-gw06 kernel: LustreError: 4509:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 18:15:28 oak-gw06 kernel: LustreError: 4509:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802164f6180) refcount = 2 Aug 8 18:15:28 oak-gw06 kernel: LustreError: 4509:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 18:15:28 oak-gw06 kernel: LustreError: 4509:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880111e33c00/0xf077f1a82c9b855c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb89fbb28 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 18:15:28 oak-gw06 kernel: LustreError: 4509:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 18:15:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 18:20:35 oak-gw06 kernel: LustreError: 4527:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802f72cee40) refcount = 2 Aug 8 18:20:35 oak-gw06 kernel: LustreError: 4527:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 18:20:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 18:20:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 18:25:45 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 18:25:45 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 18:25:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502241645, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880017563200/0xf077f1a82c9c4e4f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8a0b059 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 18:25:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 18:25:45 oak-gw06 kernel: LustreError: 4536:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802b53bfc00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 18:25:45 oak-gw06 kernel: LustreError: 4536:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 18:25:45 oak-gw06 kernel: LustreError: 4536:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802b53bfc00) refcount = 2 Aug 8 18:25:45 oak-gw06 kernel: LustreError: 4536:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 18:25:45 oak-gw06 kernel: LustreError: 4536:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880017563200/0xf077f1a82c9c4e4f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8a0b059 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 18:25:45 oak-gw06 kernel: LustreError: 4536:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 18:30:50 oak-gw06 kernel: LustreError: 4546:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880361f85b40) refcount = 2 Aug 8 18:30:50 oak-gw06 kernel: LustreError: 4546:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 18:30:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 18:30:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 18:35:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 18:35:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 18:35:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502242259, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8804046bfe00/0xf077f1a82c9c8d56 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8a1a41e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 18:35:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 18:41:08 oak-gw06 kernel: LustreError: 4565:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801550c6900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 18:41:08 oak-gw06 kernel: LustreError: 4565:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 18:41:08 oak-gw06 kernel: LustreError: 4565:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801550c6900) refcount = 2 Aug 8 18:41:08 oak-gw06 kernel: LustreError: 4565:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 18:41:08 oak-gw06 kernel: LustreError: 4565:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880274ac4400/0xf077f1a82c9cb526 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8a21c9f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 18:41:08 oak-gw06 kernel: LustreError: 4565:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 18:41:08 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 18:41:08 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 18:46:18 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 18:46:18 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 18:46:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502242878, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803d3b93600/0xf077f1a82c9d1eab lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8a299a3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 18:46:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 18:46:18 oak-gw06 kernel: LustreError: 4572:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88031226b000) refcount = 2 Aug 8 18:46:18 oak-gw06 kernel: LustreError: 4572:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 18:51:25 oak-gw06 kernel: LustreError: 4601:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802f5a8ff00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 18:51:25 oak-gw06 kernel: LustreError: 4601:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 18:51:25 oak-gw06 kernel: LustreError: 4601:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802f5a8ff00) refcount = 1 Aug 8 18:51:25 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 18:51:25 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 18:56:34 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 18:56:34 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 18:56:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502243494, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801590a9a00/0xf077f1a82c9e6589 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8a38dd1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 18:56:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 18:56:34 oak-gw06 kernel: LustreError: 4607:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff88041b30af00) refcount = 2 Aug 8 19:01:40 oak-gw06 kernel: LustreError: 4653:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88041b30aa80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 19:01:40 oak-gw06 kernel: LustreError: 4653:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 19:01:40 oak-gw06 kernel: LustreError: 4653:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041b30aa80) refcount = 2 Aug 8 19:01:40 oak-gw06 kernel: LustreError: 4653:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 19:01:40 oak-gw06 kernel: LustreError: 4653:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880392639a00/0xf077f1a82c9ea3d3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8a4052c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 19:01:40 oak-gw06 kernel: LustreError: 4653:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 19:01:40 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 19:01:40 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 19:06:47 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 19:06:47 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 19:06:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502244107, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880159c82200/0xf077f1a82c9ec950 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8a48253 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 19:06:47 oak-gw06 kernel: LustreError: 4661:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a4af7780) refcount = 2 Aug 8 19:06:47 oak-gw06 kernel: LustreError: 4661:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 19:06:47 oak-gw06 kernel: LustreError: 4661:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880159c82200/0xf077f1a82c9ec950 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8a48253 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 19:06:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 19:11:55 oak-gw06 kernel: LustreError: 4677:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802a707be40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 19:11:55 oak-gw06 kernel: LustreError: 4677:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 19:11:55 oak-gw06 kernel: LustreError: 4677:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a707be40) refcount = 2 Aug 8 19:11:55 oak-gw06 kernel: LustreError: 4677:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 19:11:55 oak-gw06 kernel: LustreError: 4677:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88031de21e00/0xf077f1a82c9f1187 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8a4f9f4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 19:11:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 19:11:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 19:17:03 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 19:17:03 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 19:17:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502244723, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880195982200/0xf077f1a82c9fc3e4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8a57722 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 19:17:03 oak-gw06 kernel: LustreError: 4689:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880205518b40) refcount = 2 Aug 8 19:17:03 oak-gw06 kernel: LustreError: 4689:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 19:17:03 oak-gw06 kernel: LustreError: 4689:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880195982200/0xf077f1a82c9fc3e4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8a57722 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 19:17:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 19:22:10 oak-gw06 kernel: LustreError: 4702:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88028cf8d240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 19:22:10 oak-gw06 kernel: LustreError: 4702:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 19:22:10 oak-gw06 kernel: LustreError: 4702:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88028cf8d240) refcount = 1 Aug 8 19:22:10 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 19:22:10 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 19:27:19 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 19:27:19 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 19:27:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502245339, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88013d527c00/0xf077f1a82ca0555c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8a66abd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 19:27:19 oak-gw06 kernel: LustreError: 4706:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801b77e49c0) refcount = 2 Aug 8 19:27:19 oak-gw06 kernel: LustreError: 4706:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 19:27:19 oak-gw06 kernel: LustreError: 4706:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88013d527c00/0xf077f1a82ca0555c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8a66abd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 19:27:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 19:32:27 oak-gw06 kernel: LustreError: 4716:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880101f33b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 19:32:27 oak-gw06 kernel: LustreError: 4716:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 19:32:27 oak-gw06 kernel: LustreError: 4716:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880101f33b40) refcount = 2 Aug 8 19:32:27 oak-gw06 kernel: LustreError: 4716:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 19:32:27 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 19:32:27 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 19:37:35 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 19:37:35 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 19:37:35 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502245955, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801a2db3a00/0xf077f1a82ca07338 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8a75e43 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 19:37:35 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 19:37:35 oak-gw06 kernel: LustreError: 4724:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803eb644240) refcount = 2 Aug 8 19:37:35 oak-gw06 kernel: LustreError: 4724:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 19:37:35 oak-gw06 kernel: LustreError: 4724:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801a2db3a00/0xf077f1a82ca07338 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8a75e43 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 19:37:35 oak-gw06 kernel: LustreError: 4724:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 19:42:44 oak-gw06 kernel: LustreError: 4735:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801220e8cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 19:42:44 oak-gw06 kernel: LustreError: 4735:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 19:42:44 oak-gw06 kernel: LustreError: 4735:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801220e8cc0) refcount = 2 Aug 8 19:42:44 oak-gw06 kernel: LustreError: 4735:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 19:42:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 19:42:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 19:47:50 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 19:47:50 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 19:47:50 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502246570, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88026839fc00/0xf077f1a82ca0a668 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8a85366 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 19:47:50 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 19:47:50 oak-gw06 kernel: LustreError: 4743:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041d9a5300) refcount = 2 Aug 8 19:47:50 oak-gw06 kernel: LustreError: 4743:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 19:47:50 oak-gw06 kernel: LustreError: 4743:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88026839fc00/0xf077f1a82ca0a668 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8a85366 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 19:47:50 oak-gw06 kernel: LustreError: 4743:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 19:52:58 oak-gw06 kernel: LustreError: 4754:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e46d6540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 19:52:58 oak-gw06 kernel: LustreError: 4754:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 19:52:58 oak-gw06 kernel: LustreError: 4754:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e46d6540) refcount = 2 Aug 8 19:52:58 oak-gw06 kernel: LustreError: 4754:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 19:52:58 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 19:52:58 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 19:58:05 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 19:58:05 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 19:58:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502247185, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802e8563600/0xf077f1a82ca105e4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8a94732 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 19:58:05 oak-gw06 kernel: LustreError: 4759:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88011dc6ca80) refcount = 2 Aug 8 19:58:05 oak-gw06 kernel: LustreError: 4759:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 19:58:05 oak-gw06 kernel: LustreError: 4759:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802e8563600/0xf077f1a82ca105e4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8a94732 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 19:58:05 oak-gw06 kernel: LustreError: 4759:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 19:58:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 20:03:12 oak-gw06 kernel: LustreError: 4807:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802113c8e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 20:03:12 oak-gw06 kernel: LustreError: 4807:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 20:03:12 oak-gw06 kernel: LustreError: 4807:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802113c8e40) refcount = 2 Aug 8 20:03:12 oak-gw06 kernel: LustreError: 4807:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 20:03:12 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 20:03:12 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 20:08:20 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 20:08:20 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 20:08:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502247800, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801123aae00/0xf077f1a82ca13ae2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8aa3bad expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 20:08:20 oak-gw06 kernel: LustreError: 4811:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800469c8b40) refcount = 2 Aug 8 20:08:20 oak-gw06 kernel: LustreError: 4811:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 20:08:20 oak-gw06 kernel: LustreError: 4811:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801123aae00/0xf077f1a82ca13ae2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8aa3bad expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 20:08:20 oak-gw06 kernel: LustreError: 4811:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 20:08:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 20:13:27 oak-gw06 kernel: LustreError: 4852:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880413e69600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 20:13:27 oak-gw06 kernel: LustreError: 4852:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 20:13:27 oak-gw06 kernel: LustreError: 4852:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e69600) refcount = 2 Aug 8 20:13:27 oak-gw06 kernel: LustreError: 4852:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 20:13:27 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 20:13:27 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 20:18:34 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 20:18:34 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 20:18:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502248414, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803f5b00e00/0xf077f1a82ca18622 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8ab2f25 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 20:18:34 oak-gw06 kernel: LustreError: 4877:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88023f4a10c0) refcount = 2 Aug 8 20:18:34 oak-gw06 kernel: LustreError: 4877:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 20:18:34 oak-gw06 kernel: LustreError: 4877:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803f5b00e00/0xf077f1a82ca18622 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8ab2f25 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 20:18:34 oak-gw06 kernel: LustreError: 4877:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 20:18:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 20:23:41 oak-gw06 kernel: LustreError: 4893:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88042d0c5f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 20:23:41 oak-gw06 kernel: LustreError: 4893:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 20:23:41 oak-gw06 kernel: LustreError: 4893:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042d0c5f00) refcount = 2 Aug 8 20:23:41 oak-gw06 kernel: LustreError: 4893:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 20:23:41 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 20:23:41 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 20:28:46 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 20:28:46 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 20:28:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502249026, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88000941e000/0xf077f1a82ca1d965 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8ac22f8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 20:28:46 oak-gw06 kernel: LustreError: 4897:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042d0c53c0) refcount = 2 Aug 8 20:28:46 oak-gw06 kernel: LustreError: 4897:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 20:28:46 oak-gw06 kernel: LustreError: 4897:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88000941e000/0xf077f1a82ca1d965 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8ac22f8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 20:28:46 oak-gw06 kernel: LustreError: 4897:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 20:28:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 20:33:54 oak-gw06 kernel: LustreError: 4909:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803b22ba840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 20:33:54 oak-gw06 kernel: LustreError: 4909:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 20:33:54 oak-gw06 kernel: LustreError: 4909:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803b22ba840) refcount = 2 Aug 8 20:33:54 oak-gw06 kernel: LustreError: 4909:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 20:33:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 20:33:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 20:39:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 20:39:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 20:39:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502249640, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801d2d42400/0xf077f1a82ca228f0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8ad1718 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 20:39:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 20:39:00 oak-gw06 kernel: LustreError: 4918:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802b1e9cc00) refcount = 2 Aug 8 20:39:00 oak-gw06 kernel: LustreError: 4918:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 20:39:00 oak-gw06 kernel: LustreError: 4918:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801d2d42400/0xf077f1a82ca228f0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8ad1718 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 20:39:00 oak-gw06 kernel: LustreError: 4918:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 20:44:07 oak-gw06 kernel: LustreError: 4932:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803c0f8a6c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 20:44:07 oak-gw06 kernel: LustreError: 4932:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 20:44:07 oak-gw06 kernel: LustreError: 4932:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803c0f8a6c0) refcount = 2 Aug 8 20:44:07 oak-gw06 kernel: LustreError: 4932:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 20:44:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 20:44:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 20:49:17 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 20:49:17 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 20:49:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502250257, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880344e34000/0xf077f1a82ca274df lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8ae0d68 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 20:49:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 20:49:17 oak-gw06 kernel: LustreError: 4940:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803c0f8a6c0) refcount = 2 Aug 8 20:49:17 oak-gw06 kernel: LustreError: 4940:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 20:49:17 oak-gw06 kernel: LustreError: 4940:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880344e34000/0xf077f1a82ca274df lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8ae0d68 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 20:49:17 oak-gw06 kernel: LustreError: 4940:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 20:54:24 oak-gw06 kernel: LustreError: 4955:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88028a616d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 20:54:24 oak-gw06 kernel: LustreError: 4955:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 20:54:24 oak-gw06 kernel: LustreError: 4955:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88028a616d80) refcount = 2 Aug 8 20:54:24 oak-gw06 kernel: LustreError: 4955:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 20:54:24 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 20:54:24 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 20:59:33 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 20:59:33 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 20:59:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502250873, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015f7d1800/0xf077f1a82ca2f28b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8af0214 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 20:59:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 20:59:33 oak-gw06 kernel: LustreError: 4963:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a3543840) refcount = 2 Aug 8 20:59:33 oak-gw06 kernel: LustreError: 4963:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 20:59:33 oak-gw06 kernel: LustreError: 4963:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015f7d1800/0xf077f1a82ca2f28b lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8af0214 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 20:59:33 oak-gw06 kernel: LustreError: 4963:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 21:04:41 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 21:04:41 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 21:09:48 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 21:09:48 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 21:09:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502251488, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801f24eb800/0xf077f1a82ca3e156 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8aff5f5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 21:09:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 21:09:48 oak-gw06 kernel: LustreError: 5018:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880419b95c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 21:09:48 oak-gw06 kernel: LustreError: 5018:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 21:09:48 oak-gw06 kernel: LustreError: 5018:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880419b95c00) refcount = 2 Aug 8 21:09:48 oak-gw06 kernel: LustreError: 5018:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 21:09:48 oak-gw06 kernel: LustreError: 5018:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801f24eb800/0xf077f1a82ca3e156 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8aff5f5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 21:14:54 oak-gw06 kernel: LustreError: 5029:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88028d31da80) refcount = 2 Aug 8 21:14:54 oak-gw06 kernel: LustreError: 5029:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 21:14:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 21:14:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 21:20:04 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 21:20:04 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 21:20:04 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502252104, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802661ebc00/0xf077f1a82ca42cf1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8b0eab6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 21:20:04 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 21:20:04 oak-gw06 kernel: LustreError: 5043:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88010da45900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 21:20:04 oak-gw06 kernel: LustreError: 5043:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 21:20:04 oak-gw06 kernel: LustreError: 5043:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88010da45900) refcount = 2 Aug 8 21:20:04 oak-gw06 kernel: LustreError: 5043:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 21:20:04 oak-gw06 kernel: LustreError: 5043:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802661ebc00/0xf077f1a82ca42cf1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8b0eab6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 21:20:04 oak-gw06 kernel: LustreError: 5043:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 21:25:12 oak-gw06 kernel: LustreError: 5051:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88012c3136c0) refcount = 2 Aug 8 21:25:12 oak-gw06 kernel: LustreError: 5051:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 21:25:12 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 21:25:12 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 21:30:20 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 21:30:20 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 21:30:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502252720, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88029d26a600/0xf077f1a82ca4e8a8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8b1dfaf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 21:30:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 21:30:20 oak-gw06 kernel: LustreError: 5076:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88017370df00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 21:30:20 oak-gw06 kernel: LustreError: 5076:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 21:30:20 oak-gw06 kernel: LustreError: 5076:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88017370df00) refcount = 2 Aug 8 21:30:20 oak-gw06 kernel: LustreError: 5076:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 21:30:20 oak-gw06 kernel: LustreError: 5076:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88029d26a600/0xf077f1a82ca4e8a8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8b1dfaf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 21:30:20 oak-gw06 kernel: LustreError: 5076:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 21:35:28 oak-gw06 kernel: LustreError: 5088:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022e65b240) refcount = 2 Aug 8 21:35:28 oak-gw06 kernel: LustreError: 5088:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 21:35:28 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 21:35:28 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 21:40:33 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 21:40:33 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 21:40:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502253333, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88028a2a0a00/0xf077f1a82ca6204b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8b2d366 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 21:40:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 21:40:33 oak-gw06 kernel: LustreError: 5099:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88029367d9c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 21:40:33 oak-gw06 kernel: LustreError: 5099:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 21:40:33 oak-gw06 kernel: LustreError: 5099:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88029367d9c0) refcount = 2 Aug 8 21:40:33 oak-gw06 kernel: LustreError: 5099:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 21:40:33 oak-gw06 kernel: LustreError: 5099:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88028a2a0a00/0xf077f1a82ca6204b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8b2d366 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 21:40:33 oak-gw06 kernel: LustreError: 5099:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 21:45:43 oak-gw06 kernel: LustreError: 5103:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801e63d4900) refcount = 2 Aug 8 21:45:43 oak-gw06 kernel: LustreError: 5103:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 21:45:43 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 21:45:43 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 21:50:53 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 21:50:53 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 21:50:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502253953, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803a861b600/0xf077f1a82ca64f29 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8b3ca1f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 21:50:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 21:56:03 oak-gw06 kernel: LustreError: 5121:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803e7f7c480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 21:56:03 oak-gw06 kernel: LustreError: 5121:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 21:56:03 oak-gw06 kernel: LustreError: 5121:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e7f7c480) refcount = 2 Aug 8 21:56:03 oak-gw06 kernel: LustreError: 5121:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 21:56:03 oak-gw06 kernel: LustreError: 5121:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801ad98f400/0xf077f1a82ca6659c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8b4432c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 21:56:03 oak-gw06 kernel: LustreError: 5121:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 21:56:03 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 21:56:03 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 22:01:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 22:01:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 22:01:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502254571, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88010681a800/0xf077f1a82ca6775b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8b4c117 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 22:01:11 oak-gw06 kernel: LustreError: 5164:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880168a16b40) refcount = 2 Aug 8 22:01:11 oak-gw06 kernel: LustreError: 5164:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 22:01:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 22:06:18 oak-gw06 kernel: LustreError: 5168:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880039ded540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 22:06:18 oak-gw06 kernel: LustreError: 5168:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 22:06:18 oak-gw06 kernel: LustreError: 5168:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880039ded540) refcount = 2 Aug 8 22:06:18 oak-gw06 kernel: LustreError: 5168:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 22:06:18 oak-gw06 kernel: LustreError: 5168:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88010681a800/0xf077f1a82ca68db9 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8b53872 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 22:06:18 oak-gw06 kernel: LustreError: 5168:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 22:06:18 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 22:06:18 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 22:11:24 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 22:11:24 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 22:11:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502255184, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802253af000/0xf077f1a82ca6b4fd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8b5b58b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 22:11:24 oak-gw06 kernel: LustreError: 5183:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88030d830300) refcount = 2 Aug 8 22:11:24 oak-gw06 kernel: LustreError: 5183:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 22:11:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 22:16:34 oak-gw06 kernel: LustreError: 5187:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802adec06c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 22:16:34 oak-gw06 kernel: LustreError: 5187:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 22:16:34 oak-gw06 kernel: LustreError: 5187:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802adec06c0) refcount = 2 Aug 8 22:16:34 oak-gw06 kernel: LustreError: 5187:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 22:16:34 oak-gw06 kernel: LustreError: 5187:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88029563ee00/0xf077f1a82ca6dbe6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8b62d25 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 22:16:34 oak-gw06 kernel: LustreError: 5187:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 22:16:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 22:16:34 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 22:21:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 22:21:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 22:21:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502255802, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801ee6c7400/0xf077f1a82ca7018d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8b6aa53 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 22:21:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 22:21:42 oak-gw06 kernel: LustreError: 5202:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803a53dbd80) refcount = 2 Aug 8 22:21:42 oak-gw06 kernel: LustreError: 5202:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 22:26:49 oak-gw06 kernel: LustreError: 5207:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801123e5b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 22:26:49 oak-gw06 kernel: LustreError: 5207:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 22:26:49 oak-gw06 kernel: LustreError: 5207:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801123e5b40) refcount = 2 Aug 8 22:26:49 oak-gw06 kernel: LustreError: 5207:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 22:26:49 oak-gw06 kernel: LustreError: 5207:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880160738000/0xf077f1a82ca72176 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8b720d5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 22:26:49 oak-gw06 kernel: LustreError: 5207:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 22:26:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 22:26:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 22:31:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 22:31:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 22:31:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502256419, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88024ce7f000/0xf077f1a82ca741c1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8b79e34 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 22:31:59 oak-gw06 kernel: LustreError: 5218:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bb641240) refcount = 2 Aug 8 22:31:59 oak-gw06 kernel: LustreError: 5218:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 22:31:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 22:37:04 oak-gw06 kernel: LustreError: 5226:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801f47a4180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 22:37:04 oak-gw06 kernel: LustreError: 5226:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 22:37:04 oak-gw06 kernel: LustreError: 5226:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801f47a4180) refcount = 2 Aug 8 22:37:04 oak-gw06 kernel: LustreError: 5226:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 22:37:04 oak-gw06 kernel: LustreError: 5226:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801d26b1200/0xf077f1a82ca7678b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8b81534 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 22:37:04 oak-gw06 kernel: LustreError: 5226:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 22:37:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 22:37:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 22:42:10 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 22:42:10 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 22:42:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502257030, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88018aaa1000/0xf077f1a82ca78bdb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8b892d2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 22:42:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 22:42:10 oak-gw06 kernel: LustreError: 5240:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801f03b7480) refcount = 2 Aug 8 22:42:10 oak-gw06 kernel: LustreError: 5240:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 22:47:20 oak-gw06 kernel: LustreError: 5250:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880210797cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 22:47:20 oak-gw06 kernel: LustreError: 5250:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 22:47:20 oak-gw06 kernel: LustreError: 5250:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880210797cc0) refcount = 2 Aug 8 22:47:20 oak-gw06 kernel: LustreError: 5250:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 22:47:20 oak-gw06 kernel: LustreError: 5250:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88031b15e200/0xf077f1a82ca7b388 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8b90b0d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 22:47:20 oak-gw06 kernel: LustreError: 5250:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 22:47:20 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 22:47:20 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 22:52:26 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 22:52:26 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 22:52:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502257646, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88031b15e200/0xf077f1a82ca7dc46 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8b987fc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 22:52:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 22:52:26 oak-gw06 kernel: LustreError: 5261:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022d4f1a80) refcount = 2 Aug 8 22:52:26 oak-gw06 kernel: LustreError: 5261:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 22:57:32 oak-gw06 kernel: LustreError: 5266:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff88027bb9a180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 22:57:32 oak-gw06 kernel: LustreError: 5266:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 22:57:32 oak-gw06 kernel: LustreError: 5266:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff88027bb9a180) refcount = 2 Aug 8 22:57:32 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 22:57:32 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 23:02:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 23:02:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 23:02:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502258262, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801a2db0e00/0xf077f1a82ca82285 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8ba7c1c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 23:02:42 oak-gw06 kernel: LustreError: 5321:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880203709d80) refcount = 2 Aug 8 23:02:42 oak-gw06 kernel: LustreError: 5321:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 23:02:42 oak-gw06 kernel: LustreError: 5321:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801a2db0e00/0xf077f1a82ca82285 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8ba7c1c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 23:02:42 oak-gw06 kernel: LustreError: 5321:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 23:02:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 23:07:52 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 23:07:52 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 23:13:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 23:13:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 23:13:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502258882, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88013d525400/0xf077f1a82ca92efb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8bb71bd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 23:13:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 23:13:02 oak-gw06 kernel: LustreError: 5339:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880039ded0c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 23:13:02 oak-gw06 kernel: LustreError: 5339:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 23:13:02 oak-gw06 kernel: LustreError: 5339:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880039ded0c0) refcount = 2 Aug 8 23:13:02 oak-gw06 kernel: LustreError: 5339:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 23:13:02 oak-gw06 kernel: LustreError: 5339:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88013d525400/0xf077f1a82ca92efb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8bb71bd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 23:18:07 oak-gw06 kernel: LustreError: 5346:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803dafb76c0) refcount = 2 Aug 8 23:18:07 oak-gw06 kernel: LustreError: 5346:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 23:18:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 23:18:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 23:23:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 23:23:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 23:23:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502259493, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801b7c5ba00/0xf077f1a82ca9d327 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8bc6440 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 23:23:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 23:23:13 oak-gw06 kernel: LustreError: 5367:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802a0748780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 23:23:13 oak-gw06 kernel: LustreError: 5367:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 8 23:23:13 oak-gw06 kernel: LustreError: 5367:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a0748780) refcount = 2 Aug 8 23:23:13 oak-gw06 kernel: LustreError: 5367:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 23:23:13 oak-gw06 kernel: LustreError: 5367:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801b7c5ba00/0xf077f1a82ca9d327 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8bc6440 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 23:23:13 oak-gw06 kernel: LustreError: 5367:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 8 23:28:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 23:28:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 23:33:30 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 23:33:30 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 23:33:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502260110, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801f8bcc400/0xf077f1a82caab59d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8bd58c9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 23:33:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 23:38:38 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 23:38:38 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 23:43:48 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 23:43:48 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 23:43:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502260728, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880165c74600/0xf077f1a82cab4b21 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8be4e71 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 23:43:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 23:43:48 oak-gw06 kernel: LustreError: 5408:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803b230a900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 23:43:48 oak-gw06 kernel: LustreError: 5408:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803b230a900) refcount = 2 Aug 8 23:43:48 oak-gw06 kernel: LustreError: 5408:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 23:43:48 oak-gw06 kernel: LustreError: 5408:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880165c74600/0xf077f1a82cab4b21 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8be4e71 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 23:48:55 oak-gw06 kernel: LustreError: 5421:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801af4ea9c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 23:48:55 oak-gw06 kernel: LustreError: 5421:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801af4ea9c0) refcount = 2 Aug 8 23:48:55 oak-gw06 kernel: LustreError: 5421:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 23:48:55 oak-gw06 kernel: LustreError: 5421:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801b916f800/0xf077f1a82cab8c3c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8bec6ac expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 23:48:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 23:48:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 8 23:54:03 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 8 23:54:03 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 8 23:54:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502261343, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880221d4da00/0xf077f1a82cac0eaa lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8bf4451 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 23:54:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 8 23:54:03 oak-gw06 kernel: LustreError: 5453:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880162760540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 23:54:03 oak-gw06 kernel: LustreError: 5453:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880162760540) refcount = 2 Aug 8 23:54:03 oak-gw06 kernel: LustreError: 5453:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 23:54:03 oak-gw06 kernel: LustreError: 5453:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880221d4da00/0xf077f1a82cac0eaa lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8bf4451 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 23:59:11 oak-gw06 kernel: LustreError: 5506:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802bde38540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 8 23:59:11 oak-gw06 kernel: LustreError: 5506:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bde38540) refcount = 2 Aug 8 23:59:11 oak-gw06 kernel: LustreError: 5506:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 8 23:59:11 oak-gw06 kernel: LustreError: 5506:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88013bff4200/0xf077f1a82cacc83f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8bfbb7b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 8 23:59:11 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 8 23:59:11 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 00:04:20 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 00:04:20 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 00:04:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502261960, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801dcdc7600/0xf077f1a82cb12ef9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8c038c5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 00:04:20 oak-gw06 kernel: LustreError: 5555:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880396b1c180) refcount = 2 Aug 9 00:04:20 oak-gw06 kernel: LustreError: 5555:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 00:04:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 00:09:29 oak-gw06 kernel: LustreError: 5558:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880396b1c000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 00:09:29 oak-gw06 kernel: LustreError: 5558:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 00:09:29 oak-gw06 kernel: LustreError: 5558:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880396b1c000) refcount = 2 Aug 9 00:09:29 oak-gw06 kernel: LustreError: 5558:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 00:09:29 oak-gw06 kernel: LustreError: 5558:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802d010d400/0xf077f1a82cb14f4b lrc: 3/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8c0b04a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 00:09:29 oak-gw06 kernel: LustreError: 5558:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 00:09:29 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 00:09:29 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 00:14:36 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 00:14:36 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 00:14:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502262576, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802f8ea7000/0xf077f1a82cb16fce lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8c12d9b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 00:14:36 oak-gw06 kernel: LustreError: 5570:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800afc58300) refcount = 2 Aug 9 00:14:36 oak-gw06 kernel: LustreError: 5570:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 00:14:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 00:19:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 00:19:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 00:24:54 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 00:24:54 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 00:24:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502263194, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880250bc4600/0xf077f1a82cb1af1b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8c22351 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 00:24:54 oak-gw06 kernel: LustreError: 5591:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88021070fd80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 00:24:54 oak-gw06 kernel: LustreError: 5591:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 00:24:54 oak-gw06 kernel: LustreError: 5591:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88021070fd80) refcount = 2 Aug 9 00:24:54 oak-gw06 kernel: LustreError: 5591:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 00:24:54 oak-gw06 kernel: LustreError: 5591:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880250bc4600/0xf077f1a82cb1af1b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8c22351 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 00:24:54 oak-gw06 kernel: LustreError: 5591:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 00:24:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 00:30:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 00:30:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 00:35:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 00:35:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 00:35:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502263811, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880411ac5800/0xf077f1a82cb3e6b5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8c31858 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 00:35:11 oak-gw06 kernel: LustreError: 5629:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ee8a56c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 00:35:11 oak-gw06 kernel: LustreError: 5629:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ee8a56c0) refcount = 2 Aug 9 00:35:11 oak-gw06 kernel: LustreError: 5629:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 00:35:11 oak-gw06 kernel: LustreError: 5629:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880411ac5800/0xf077f1a82cb3e6b5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8c31858 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 00:35:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 00:40:17 oak-gw06 kernel: LustreError: 5643:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880205bf4540) refcount = 2 Aug 9 00:40:17 oak-gw06 kernel: LustreError: 5643:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 00:40:17 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 00:40:17 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 00:45:26 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 00:45:26 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 00:45:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502264426, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802c64c6600/0xf077f1a82cb448ae lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8c40d66 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 00:45:26 oak-gw06 kernel: LustreError: 5646:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801c653c540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 00:45:26 oak-gw06 kernel: LustreError: 5646:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 00:45:26 oak-gw06 kernel: LustreError: 5646:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c653c540) refcount = 2 Aug 9 00:45:26 oak-gw06 kernel: LustreError: 5646:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 00:45:26 oak-gw06 kernel: LustreError: 5646:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802c64c6600/0xf077f1a82cb448ae lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8c40d66 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 00:45:26 oak-gw06 kernel: LustreError: 5646:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 00:45:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 00:50:33 oak-gw06 kernel: LustreError: 5659:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff88020a4aa240) refcount = 2 Aug 9 00:50:33 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 00:50:33 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 00:55:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 00:55:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 00:55:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502265042, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803eaac6c00/0xf077f1a82cb48ea0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8c5024a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 00:55:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 00:55:42 oak-gw06 kernel: LustreError: 5667:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88020a4aa180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 00:55:42 oak-gw06 kernel: LustreError: 5667:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 00:55:42 oak-gw06 kernel: LustreError: 5667:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88020a4aa180) refcount = 2 Aug 9 00:55:42 oak-gw06 kernel: LustreError: 5667:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 00:55:42 oak-gw06 kernel: LustreError: 5667:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803eaac6c00/0xf077f1a82cb48ea0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8c5024a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 01:00:50 oak-gw06 kernel: LustreError: 5745:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880162760480) refcount = 2 Aug 9 01:00:50 oak-gw06 kernel: LustreError: 5745:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 01:00:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 01:00:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 01:05:55 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 01:05:55 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 01:05:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502265655, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802c6663000/0xf077f1a82cbb418b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8c5f77b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 01:05:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 01:05:55 oak-gw06 kernel: LustreError: 5880:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88020e617300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 01:05:55 oak-gw06 kernel: LustreError: 5880:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 01:05:55 oak-gw06 kernel: LustreError: 5880:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88020e617300) refcount = 2 Aug 9 01:05:55 oak-gw06 kernel: LustreError: 5880:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 01:05:55 oak-gw06 kernel: LustreError: 5880:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802c6663000/0xf077f1a82cbb418b lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8c5f77b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 01:05:55 oak-gw06 kernel: LustreError: 5880:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 01:11:03 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 01:11:03 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 01:16:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 01:16:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 01:16:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502266271, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880210684e00/0xf077f1a82cd1652a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8c6ec04 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 01:16:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 01:16:11 oak-gw06 kernel: LustreError: 6039:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88029429a0c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 01:16:11 oak-gw06 kernel: LustreError: 6039:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88029429a0c0) refcount = 2 Aug 9 01:16:11 oak-gw06 kernel: LustreError: 6039:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 01:16:11 oak-gw06 kernel: LustreError: 6039:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880210684e00/0xf077f1a82cd1652a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8c6ec04 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 01:21:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 01:21:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 01:26:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 01:26:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 01:26:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502266891, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88028729a600/0xf077f1a82cd1aac8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8c7e1b3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 01:26:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 01:26:31 oak-gw06 kernel: LustreError: 6058:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88014f64bcc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 01:26:31 oak-gw06 kernel: LustreError: 6058:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88014f64bcc0) refcount = 2 Aug 9 01:26:31 oak-gw06 kernel: LustreError: 6058:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 01:26:31 oak-gw06 kernel: LustreError: 6058:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88028729a600/0xf077f1a82cd1aac8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8c7e1b3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 01:31:39 oak-gw06 kernel: LustreError: 6081:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042b490e40) refcount = 2 Aug 9 01:31:39 oak-gw06 kernel: LustreError: 6081:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 01:31:39 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 01:31:39 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 01:36:46 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 01:36:46 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 01:36:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502267506, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88036be72c00/0xf077f1a82cd31c39 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8c8d6cf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 01:36:46 oak-gw06 kernel: LustreError: 6094:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801a1262600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 01:36:46 oak-gw06 kernel: LustreError: 6094:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 01:36:46 oak-gw06 kernel: LustreError: 6094:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801a1262600) refcount = 2 Aug 9 01:36:46 oak-gw06 kernel: LustreError: 6094:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 01:36:46 oak-gw06 kernel: LustreError: 6094:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88036be72c00/0xf077f1a82cd31c39 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8c8d6cf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 01:36:46 oak-gw06 kernel: LustreError: 6094:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 01:36:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 01:41:53 oak-gw06 kernel: LustreError: 6108:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880158c24d80) refcount = 2 Aug 9 01:41:53 oak-gw06 kernel: LustreError: 6108:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 01:41:53 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 01:41:53 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 01:47:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 01:47:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 01:47:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502268122, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880155151200/0xf077f1a82cd438bc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8c9cc5b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 01:47:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 01:47:02 oak-gw06 kernel: LustreError: 6113:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880284b5fc00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 01:47:02 oak-gw06 kernel: LustreError: 6113:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 01:47:02 oak-gw06 kernel: LustreError: 6113:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880284b5fc00) refcount = 2 Aug 9 01:47:02 oak-gw06 kernel: LustreError: 6113:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 01:47:02 oak-gw06 kernel: LustreError: 6113:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880155151200/0xf077f1a82cd438bc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8c9cc5b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 01:47:02 oak-gw06 kernel: LustreError: 6113:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 01:52:08 oak-gw06 kernel: LustreError: 6124:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88017845a300) refcount = 2 Aug 9 01:52:08 oak-gw06 kernel: LustreError: 6124:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 01:52:08 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 01:52:08 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 01:57:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 01:57:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 01:57:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502268736, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88024be1da00/0xf077f1a82cd45946 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8cac1a1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 01:57:16 oak-gw06 kernel: LustreError: 6132:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800222966c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 01:57:16 oak-gw06 kernel: LustreError: 6132:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 01:57:16 oak-gw06 kernel: LustreError: 6132:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800222966c0) refcount = 2 Aug 9 01:57:16 oak-gw06 kernel: LustreError: 6132:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 01:57:16 oak-gw06 kernel: LustreError: 6132:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88024be1da00/0xf077f1a82cd45946 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8cac1a1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 01:57:16 oak-gw06 kernel: LustreError: 6132:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 01:57:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 02:02:26 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 02:02:26 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 02:07:33 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 02:07:33 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 02:07:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502269353, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88013f497200/0xf077f1a82cd4f0c2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8cbb814 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 02:07:33 oak-gw06 kernel: LustreError: 6189:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88035d2ed600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 02:07:33 oak-gw06 kernel: LustreError: 6189:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88035d2ed600) refcount = 2 Aug 9 02:07:33 oak-gw06 kernel: LustreError: 6189:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 02:07:33 oak-gw06 kernel: LustreError: 6189:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88013f497200/0xf077f1a82cd4f0c2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8cbb814 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 02:07:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 02:12:42 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 02:12:42 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 02:17:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 02:17:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 02:17:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502269969, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880159660a00/0xf077f1a82cd53f7b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8ccadd1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 02:17:49 oak-gw06 kernel: LustreError: 6202:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801c5e74b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 02:17:49 oak-gw06 kernel: LustreError: 6202:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c5e74b40) refcount = 2 Aug 9 02:17:49 oak-gw06 kernel: LustreError: 6202:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 02:17:49 oak-gw06 kernel: LustreError: 6202:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880159660a00/0xf077f1a82cd53f7b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8ccadd1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 02:17:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 02:22:58 oak-gw06 kernel: LustreError: 6224:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88032482d780) refcount = 2 Aug 9 02:22:58 oak-gw06 kernel: LustreError: 6224:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 02:22:58 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 02:22:58 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 02:28:05 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 02:28:05 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 02:28:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502270585, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801f3241a00/0xf077f1a82cd62bec lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8cda2bc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 02:28:05 oak-gw06 kernel: LustreError: 6229:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880046de4b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 02:28:05 oak-gw06 kernel: LustreError: 6229:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 02:28:05 oak-gw06 kernel: LustreError: 6229:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880046de4b40) refcount = 2 Aug 9 02:28:05 oak-gw06 kernel: LustreError: 6229:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 02:28:05 oak-gw06 kernel: LustreError: 6229:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801f3241a00/0xf077f1a82cd62bec lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8cda2bc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 02:28:05 oak-gw06 kernel: LustreError: 6229:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 02:28:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 02:33:14 oak-gw06 kernel: LustreError: 6245:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880412fba240) refcount = 2 Aug 9 02:33:14 oak-gw06 kernel: LustreError: 6245:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 02:33:14 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 02:33:14 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 02:38:25 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 02:38:25 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 02:38:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502271205, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88026485de00/0xf077f1a82cd6ba62 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8ce990c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 02:38:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 02:38:25 oak-gw06 kernel: LustreError: 6249:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88020edf0a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 02:38:25 oak-gw06 kernel: LustreError: 6249:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 02:38:25 oak-gw06 kernel: LustreError: 6249:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88020edf0a80) refcount = 2 Aug 9 02:38:25 oak-gw06 kernel: LustreError: 6249:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 02:38:25 oak-gw06 kernel: LustreError: 6249:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88026485de00/0xf077f1a82cd6ba62 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8ce990c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 02:38:25 oak-gw06 kernel: LustreError: 6249:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 02:43:31 oak-gw06 kernel: LustreError: 6260:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88027c3de3c0) refcount = 2 Aug 9 02:43:31 oak-gw06 kernel: LustreError: 6260:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 02:43:31 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 02:43:31 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 02:48:40 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 02:48:40 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 02:48:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502271820, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88028bb9a200/0xf077f1a82cd6d7ea lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8cf8e05 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 02:48:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 02:48:40 oak-gw06 kernel: LustreError: 6268:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88029069d000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 02:48:40 oak-gw06 kernel: LustreError: 6268:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 02:48:40 oak-gw06 kernel: LustreError: 6268:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88029069d000) refcount = 2 Aug 9 02:48:40 oak-gw06 kernel: LustreError: 6268:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 02:48:40 oak-gw06 kernel: LustreError: 6268:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88028bb9a200/0xf077f1a82cd6d7ea lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8cf8e05 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 02:48:40 oak-gw06 kernel: LustreError: 6268:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 02:53:46 oak-gw06 kernel: LustreError: 6280:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803b2217540) refcount = 2 Aug 9 02:53:46 oak-gw06 kernel: LustreError: 6280:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 02:53:46 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 02:53:46 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 02:58:54 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 02:58:54 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 02:58:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502272434, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801fbba8a00/0xf077f1a82cd7130f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8d0834b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 02:58:54 oak-gw06 kernel: LustreError: 6282:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88017845a9c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 02:58:54 oak-gw06 kernel: LustreError: 6282:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 02:58:54 oak-gw06 kernel: LustreError: 6282:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88017845a9c0) refcount = 2 Aug 9 02:58:54 oak-gw06 kernel: LustreError: 6282:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 02:58:54 oak-gw06 kernel: LustreError: 6282:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801fbba8a00/0xf077f1a82cd7130f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8d0834b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 02:58:54 oak-gw06 kernel: LustreError: 6282:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 02:58:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 03:04:01 oak-gw06 kernel: LustreError: 6329:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bb2116c0) refcount = 2 Aug 9 03:04:01 oak-gw06 kernel: LustreError: 6329:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 03:04:01 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 03:04:01 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 03:09:07 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 03:09:07 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 03:09:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502273047, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88014f5ff200/0xf077f1a82cd75805 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8d177f7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 03:09:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 03:09:07 oak-gw06 kernel: LustreError: 6367:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802603a4f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 03:09:07 oak-gw06 kernel: LustreError: 6367:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 03:09:07 oak-gw06 kernel: LustreError: 6367:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802603a4f00) refcount = 2 Aug 9 03:09:07 oak-gw06 kernel: LustreError: 6367:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 03:09:07 oak-gw06 kernel: LustreError: 6367:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88014f5ff200/0xf077f1a82cd75805 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8d177f7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 03:09:07 oak-gw06 kernel: LustreError: 6367:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 03:14:14 oak-gw06 kernel: LustreError: 6376:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880013056180) refcount = 2 Aug 9 03:14:14 oak-gw06 kernel: LustreError: 6376:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 03:14:14 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 03:14:14 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 03:19:23 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 03:19:23 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 03:19:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502273663, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88009b66b000/0xf077f1a82cd79108 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8d26cd4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 03:19:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 03:19:23 oak-gw06 kernel: LustreError: 6384:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803b36e73c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 03:19:23 oak-gw06 kernel: LustreError: 6384:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 03:19:23 oak-gw06 kernel: LustreError: 6384:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803b36e73c0) refcount = 2 Aug 9 03:19:23 oak-gw06 kernel: LustreError: 6384:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 03:19:23 oak-gw06 kernel: LustreError: 6384:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88009b66b000/0xf077f1a82cd79108 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8d26cd4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 03:19:23 oak-gw06 kernel: LustreError: 6384:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 03:24:29 oak-gw06 kernel: LustreError: 6394:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88024b6b30c0) refcount = 1 Aug 9 03:24:29 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 03:24:29 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 03:29:38 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 03:29:38 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 03:29:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502274278, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88030d9ac200/0xf077f1a82cd7c5d5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8d362e5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 03:29:38 oak-gw06 kernel: LustreError: 6401:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88022edb5540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 03:29:38 oak-gw06 kernel: LustreError: 6401:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 03:29:38 oak-gw06 kernel: LustreError: 6401:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022edb5540) refcount = 2 Aug 9 03:29:38 oak-gw06 kernel: LustreError: 6401:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 03:29:38 oak-gw06 kernel: LustreError: 6401:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88030d9ac200/0xf077f1a82cd7c5d5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8d362e5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 03:29:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 03:34:46 oak-gw06 kernel: LustreError: 6417:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ef60ed80) refcount = 2 Aug 9 03:34:46 oak-gw06 kernel: LustreError: 6417:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 03:34:46 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 03:34:46 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 03:39:52 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 03:39:52 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 03:39:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502274892, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880159b6be00/0xf077f1a82cd7ff4f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8d45791 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 03:39:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 03:39:52 oak-gw06 kernel: LustreError: 6420:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880206283d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 03:39:52 oak-gw06 kernel: LustreError: 6420:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 03:39:52 oak-gw06 kernel: LustreError: 6420:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880206283d80) refcount = 2 Aug 9 03:39:52 oak-gw06 kernel: LustreError: 6420:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 03:39:52 oak-gw06 kernel: LustreError: 6420:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880159b6be00/0xf077f1a82cd7ff4f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8d45791 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 03:39:52 oak-gw06 kernel: LustreError: 6420:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 03:45:01 oak-gw06 kernel: LustreError: 6432:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880312348c00) refcount = 2 Aug 9 03:45:01 oak-gw06 kernel: LustreError: 6432:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 03:45:01 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 03:45:01 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 03:50:09 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 03:50:09 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 03:50:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502275509, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802661eb600/0xf077f1a82cd83596 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8d54d40 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 03:50:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 03:50:09 oak-gw06 kernel: LustreError: 6447:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88015e2f6840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 03:50:09 oak-gw06 kernel: LustreError: 6447:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 03:50:09 oak-gw06 kernel: LustreError: 6447:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88015e2f6840) refcount = 2 Aug 9 03:50:09 oak-gw06 kernel: LustreError: 6447:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 03:50:09 oak-gw06 kernel: LustreError: 6447:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802661eb600/0xf077f1a82cd83596 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8d54d40 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 03:50:09 oak-gw06 kernel: LustreError: 6447:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 03:55:15 oak-gw06 kernel: LustreError: 6451:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802dcac0a80) refcount = 2 Aug 9 03:55:15 oak-gw06 kernel: LustreError: 6451:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 03:55:15 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 03:55:15 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 04:00:22 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 04:00:22 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 04:00:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502276122, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880279f5d400/0xf077f1a82cd854d0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8d64113 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 04:00:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 04:00:22 oak-gw06 kernel: LustreError: 6466:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88042c79d540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 04:00:22 oak-gw06 kernel: LustreError: 6466:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 04:00:22 oak-gw06 kernel: LustreError: 6466:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042c79d540) refcount = 2 Aug 9 04:00:22 oak-gw06 kernel: LustreError: 6466:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 04:00:22 oak-gw06 kernel: LustreError: 6466:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880279f5d400/0xf077f1a82cd854d0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8d64113 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 04:00:22 oak-gw06 kernel: LustreError: 6466:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 04:05:30 oak-gw06 kernel: LustreError: 6503:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800073cfa80) refcount = 2 Aug 9 04:05:30 oak-gw06 kernel: LustreError: 6503:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 04:05:30 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 04:05:30 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 04:10:36 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 04:10:36 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 04:10:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502276736, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801ac553e00/0xf077f1a82cd8bbae lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8d7368a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 04:10:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 04:10:36 oak-gw06 kernel: LustreError: 6514:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800073cf840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 04:10:36 oak-gw06 kernel: LustreError: 6514:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 04:10:36 oak-gw06 kernel: LustreError: 6514:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800073cf840) refcount = 2 Aug 9 04:10:36 oak-gw06 kernel: LustreError: 6514:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 04:10:36 oak-gw06 kernel: LustreError: 6514:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801ac553e00/0xf077f1a82cd8bbae lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8d7368a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 04:10:36 oak-gw06 kernel: LustreError: 6514:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 04:15:46 oak-gw06 kernel: LustreError: 6519:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042c4bf540) refcount = 2 Aug 9 04:15:46 oak-gw06 kernel: LustreError: 6519:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 04:15:46 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 04:15:46 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 04:20:55 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 04:20:55 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 04:20:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502277355, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801ad98f000/0xf077f1a82cd8f0f2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8d82d82 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 04:20:55 oak-gw06 kernel: LustreError: 6534:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802bb211d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 04:20:55 oak-gw06 kernel: LustreError: 6534:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 04:20:55 oak-gw06 kernel: LustreError: 6534:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bb211d80) refcount = 2 Aug 9 04:20:55 oak-gw06 kernel: LustreError: 6534:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 04:20:55 oak-gw06 kernel: LustreError: 6534:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801ad98f000/0xf077f1a82cd8f0f2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8d82d82 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 04:20:55 oak-gw06 kernel: LustreError: 6534:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 04:20:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 04:26:01 oak-gw06 kernel: LustreError: 6542:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880428bb76c0) refcount = 2 Aug 9 04:26:01 oak-gw06 kernel: LustreError: 6542:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 04:26:01 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 04:26:01 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 04:31:09 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 04:31:09 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 04:31:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502277969, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880304317400/0xf077f1a82cd941d4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8d92315 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 04:31:09 oak-gw06 kernel: LustreError: 6555:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88001d6c9b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 04:31:09 oak-gw06 kernel: LustreError: 6555:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 04:31:09 oak-gw06 kernel: LustreError: 6555:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88001d6c9b40) refcount = 2 Aug 9 04:31:09 oak-gw06 kernel: LustreError: 6555:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 04:31:09 oak-gw06 kernel: LustreError: 6555:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880304317400/0xf077f1a82cd941d4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8d92315 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 04:31:09 oak-gw06 kernel: LustreError: 6555:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 04:31:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 04:36:19 oak-gw06 kernel: LustreError: 6558:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803a46bbd80) refcount = 2 Aug 9 04:36:19 oak-gw06 kernel: LustreError: 6558:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 04:36:19 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 04:36:19 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 04:41:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 04:41:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 04:41:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502278588, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880122189e00/0xf077f1a82cd97638 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8da1a29 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 04:41:28 oak-gw06 kernel: LustreError: 6573:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880265b4ac00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 04:41:28 oak-gw06 kernel: LustreError: 6573:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 04:41:28 oak-gw06 kernel: LustreError: 6573:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880265b4ac00) refcount = 2 Aug 9 04:41:28 oak-gw06 kernel: LustreError: 6573:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 04:41:28 oak-gw06 kernel: LustreError: 6573:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880122189e00/0xf077f1a82cd97638 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8da1a29 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 04:41:28 oak-gw06 kernel: LustreError: 6573:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 04:41:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 04:46:34 oak-gw06 kernel: LustreError: 6577:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880426fcad80) refcount = 2 Aug 9 04:46:34 oak-gw06 kernel: LustreError: 6577:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 04:46:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 04:46:34 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 04:51:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 04:51:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 04:51:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502279201, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88029def8e00/0xf077f1a82cd9e590 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8db0fc3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 04:51:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 04:51:41 oak-gw06 kernel: LustreError: 6592:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800073cf480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 04:51:41 oak-gw06 kernel: LustreError: 6592:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 04:51:41 oak-gw06 kernel: LustreError: 6592:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800073cf480) refcount = 2 Aug 9 04:51:41 oak-gw06 kernel: LustreError: 6592:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 04:51:41 oak-gw06 kernel: LustreError: 6592:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88029def8e00/0xf077f1a82cd9e590 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8db0fc3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 04:51:41 oak-gw06 kernel: LustreError: 6592:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 04:56:50 oak-gw06 kernel: LustreError: 6621:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88011dc6cc00) refcount = 2 Aug 9 04:56:50 oak-gw06 kernel: LustreError: 6621:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 04:56:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 04:56:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 05:01:56 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 05:01:56 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 05:01:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502279816, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880252b1a400/0xf077f1a82cdc5a39 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8dc04c3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 05:01:56 oak-gw06 kernel: LustreError: 6668:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880036d95480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 05:01:56 oak-gw06 kernel: LustreError: 6668:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 05:01:56 oak-gw06 kernel: LustreError: 6668:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880036d95480) refcount = 2 Aug 9 05:01:56 oak-gw06 kernel: LustreError: 6668:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 05:01:56 oak-gw06 kernel: LustreError: 6668:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880252b1a400/0xf077f1a82cdc5a39 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8dc04c3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 05:01:56 oak-gw06 kernel: LustreError: 6668:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 05:01:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 05:07:04 oak-gw06 kernel: LustreError: 6675:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88021265b6c0) refcount = 2 Aug 9 05:07:04 oak-gw06 kernel: LustreError: 6675:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 05:07:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 05:07:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 05:12:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 05:12:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 05:12:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502280433, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880180b77000/0xf077f1a82cdd1b61 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8dcf9b5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 05:12:13 oak-gw06 kernel: LustreError: 6690:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802256a2600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 05:12:13 oak-gw06 kernel: LustreError: 6690:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 05:12:13 oak-gw06 kernel: LustreError: 6690:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802256a2600) refcount = 2 Aug 9 05:12:13 oak-gw06 kernel: LustreError: 6690:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 05:12:13 oak-gw06 kernel: LustreError: 6690:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880180b77000/0xf077f1a82cdd1b61 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8dcf9b5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 05:12:13 oak-gw06 kernel: LustreError: 6690:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 05:12:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 05:17:23 oak-gw06 kernel: LustreError: 6694:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413f39540) refcount = 2 Aug 9 05:17:23 oak-gw06 kernel: LustreError: 6694:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 05:17:23 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 05:17:23 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 05:22:33 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 05:22:33 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 05:22:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502281053, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802863ed200/0xf077f1a82cddc91f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8ddefb8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 05:22:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 05:22:33 oak-gw06 kernel: LustreError: 6710:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802ebd4bf00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 05:22:33 oak-gw06 kernel: LustreError: 6710:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 05:22:33 oak-gw06 kernel: LustreError: 6710:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802ebd4bf00) refcount = 2 Aug 9 05:22:33 oak-gw06 kernel: LustreError: 6710:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 05:22:33 oak-gw06 kernel: LustreError: 6710:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802863ed200/0xf077f1a82cddc91f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8ddefb8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 05:22:33 oak-gw06 kernel: LustreError: 6710:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 05:27:42 oak-gw06 kernel: LustreError: 6717:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880374aead80) refcount = 2 Aug 9 05:27:42 oak-gw06 kernel: LustreError: 6717:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 05:27:42 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 05:27:42 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 05:32:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 05:32:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 05:32:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502281669, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880288a00e00/0xf077f1a82cde4fae lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8dee5de expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 05:32:49 oak-gw06 kernel: LustreError: 6734:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ac5c0a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 05:32:49 oak-gw06 kernel: LustreError: 6734:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 05:32:49 oak-gw06 kernel: LustreError: 6734:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ac5c0a80) refcount = 2 Aug 9 05:32:49 oak-gw06 kernel: LustreError: 6734:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 05:32:49 oak-gw06 kernel: LustreError: 6734:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880288a00e00/0xf077f1a82cde4fae lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8dee5de expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 05:32:49 oak-gw06 kernel: LustreError: 6734:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 05:32:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 05:37:58 oak-gw06 kernel: LustreError: 6742:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a3543cc0) refcount = 2 Aug 9 05:37:58 oak-gw06 kernel: LustreError: 6742:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 05:37:58 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 05:37:58 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 05:43:08 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 05:43:08 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 05:43:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502282288, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803ed3aee00/0xf077f1a82cdf16da lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8dfdd93 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 05:43:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 05:43:08 oak-gw06 kernel: LustreError: 6754:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a9482f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 05:43:08 oak-gw06 kernel: LustreError: 6754:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 05:43:08 oak-gw06 kernel: LustreError: 6754:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a9482f00) refcount = 2 Aug 9 05:43:08 oak-gw06 kernel: LustreError: 6754:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 05:43:08 oak-gw06 kernel: LustreError: 6754:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803ed3aee00/0xf077f1a82cdf16da lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8dfdd93 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 05:43:08 oak-gw06 kernel: LustreError: 6754:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 05:48:16 oak-gw06 kernel: LustreError: 6762:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88011fe1d300) refcount = 2 Aug 9 05:48:16 oak-gw06 kernel: LustreError: 6762:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 05:48:16 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 05:48:16 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 05:53:22 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 05:53:22 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 05:53:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502282902, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801eb7dc400/0xf077f1a82cdfa44d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8e0d3f8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 05:53:22 oak-gw06 kernel: LustreError: 6778:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880369ff93c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 05:53:22 oak-gw06 kernel: LustreError: 6778:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 05:53:22 oak-gw06 kernel: LustreError: 6778:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880369ff93c0) refcount = 2 Aug 9 05:53:22 oak-gw06 kernel: LustreError: 6778:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 05:53:22 oak-gw06 kernel: LustreError: 6778:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801eb7dc400/0xf077f1a82cdfa44d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8e0d3f8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 05:53:22 oak-gw06 kernel: LustreError: 6778:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 05:53:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 05:58:30 oak-gw06 kernel: LustreError: 6786:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802037093c0) refcount = 2 Aug 9 05:58:30 oak-gw06 kernel: LustreError: 6786:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 05:58:30 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 05:58:30 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 06:03:37 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 06:03:37 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 06:03:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502283517, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88013be75200/0xf077f1a82ce01af9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8e1cafe expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 06:03:37 oak-gw06 kernel: LustreError: 6831:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803812db000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 06:03:37 oak-gw06 kernel: LustreError: 6831:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 06:03:37 oak-gw06 kernel: LustreError: 6831:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803812db000) refcount = 2 Aug 9 06:03:37 oak-gw06 kernel: LustreError: 6831:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 06:03:37 oak-gw06 kernel: LustreError: 6831:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88013be75200/0xf077f1a82ce01af9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8e1cafe expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 06:03:37 oak-gw06 kernel: LustreError: 6831:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 06:03:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 06:08:47 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 06:08:47 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 06:13:53 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 06:13:53 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 06:13:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502284133, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803e10f6a00/0xf077f1a82ce06e7b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8e2c067 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 06:13:53 oak-gw06 kernel: LustreError: 6861:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802a82e96c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 06:13:53 oak-gw06 kernel: LustreError: 6861:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a82e96c0) refcount = 2 Aug 9 06:13:53 oak-gw06 kernel: LustreError: 6861:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 06:13:53 oak-gw06 kernel: LustreError: 6861:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803e10f6a00/0xf077f1a82ce06e7b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8e2c067 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 06:13:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 06:19:02 oak-gw06 kernel: LustreError: 6869:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88020abf26c0) refcount = 2 Aug 9 06:19:02 oak-gw06 kernel: LustreError: 6869:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 06:19:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 06:19:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 06:24:10 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 06:24:10 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 06:24:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502284750, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88024a13da00/0xf077f1a82ce1da27 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8e3b5f3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 06:24:10 oak-gw06 kernel: LustreError: 6904:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880137be8d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 06:24:10 oak-gw06 kernel: LustreError: 6904:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 06:24:10 oak-gw06 kernel: LustreError: 6904:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880137be8d80) refcount = 2 Aug 9 06:24:10 oak-gw06 kernel: LustreError: 6904:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 06:24:10 oak-gw06 kernel: LustreError: 6904:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88024a13da00/0xf077f1a82ce1da27 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8e3b5f3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 06:24:10 oak-gw06 kernel: LustreError: 6904:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 06:24:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 06:29:19 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 06:29:19 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 06:34:26 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 06:34:26 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 06:34:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502285366, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801002d3c00/0xf077f1a82ce857c7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8e4ad07 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 06:34:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 06:34:26 oak-gw06 kernel: LustreError: 7021:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ec38ad80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 06:34:26 oak-gw06 kernel: LustreError: 7021:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ec38ad80) refcount = 2 Aug 9 06:34:26 oak-gw06 kernel: LustreError: 7021:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 06:34:26 oak-gw06 kernel: LustreError: 7021:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801002d3c00/0xf077f1a82ce857c7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8e4ad07 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 06:39:32 oak-gw06 kernel: LustreError: 7086:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802772913c0) refcount = 2 Aug 9 06:39:32 oak-gw06 kernel: LustreError: 7086:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 06:39:32 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 06:39:32 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 06:44:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 06:44:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 06:44:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502285981, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88012cb77c00/0xf077f1a82cf40289 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8e5a37a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 06:44:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 06:44:41 oak-gw06 kernel: LustreError: 7198:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880287eef600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 06:44:41 oak-gw06 kernel: LustreError: 7198:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 06:44:41 oak-gw06 kernel: LustreError: 7198:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880287eef600) refcount = 2 Aug 9 06:44:41 oak-gw06 kernel: LustreError: 7198:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 06:44:41 oak-gw06 kernel: LustreError: 7198:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88012cb77c00/0xf077f1a82cf40289 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8e5a37a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 06:44:41 oak-gw06 kernel: LustreError: 7198:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 06:49:51 oak-gw06 kernel: LustreError: 7258:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803b8bf4480) refcount = 2 Aug 9 06:49:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 06:49:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 06:54:58 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 06:54:58 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 06:54:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502286598, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88024be1d000/0xf077f1a82d028b4f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8e699a7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 06:54:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 06:54:58 oak-gw06 kernel: LustreError: 7269:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ca452d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 06:54:58 oak-gw06 kernel: LustreError: 7269:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 06:54:58 oak-gw06 kernel: LustreError: 7269:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ca452d80) refcount = 2 Aug 9 06:54:58 oak-gw06 kernel: LustreError: 7269:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 06:54:58 oak-gw06 kernel: LustreError: 7269:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88024be1d000/0xf077f1a82d028b4f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8e699a7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 07:00:07 oak-gw06 kernel: LustreError: 7288:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88037ea01300) refcount = 2 Aug 9 07:00:07 oak-gw06 kernel: LustreError: 7288:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 07:00:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 07:00:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 07:05:15 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 07:05:15 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 07:05:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502287215, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801930a9c00/0xf077f1a82d033e4d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8e78fa3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 07:05:15 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 07:05:15 oak-gw06 kernel: LustreError: 7329:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88027c3280c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 07:05:15 oak-gw06 kernel: LustreError: 7329:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 07:05:15 oak-gw06 kernel: LustreError: 7329:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88027c3280c0) refcount = 2 Aug 9 07:05:15 oak-gw06 kernel: LustreError: 7329:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 07:05:15 oak-gw06 kernel: LustreError: 7329:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801930a9c00/0xf077f1a82d033e4d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8e78fa3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 07:05:15 oak-gw06 kernel: LustreError: 7329:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 07:10:21 oak-gw06 kernel: LustreError: 7341:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88010c43de40) refcount = 2 Aug 9 07:10:21 oak-gw06 kernel: LustreError: 7341:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 07:10:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 07:10:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 07:15:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 07:15:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 07:15:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502287828, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801836c7600/0xf077f1a82d03c22e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8e88480 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 07:15:28 oak-gw06 kernel: LustreError: 7345:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802a30ca6c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 07:15:28 oak-gw06 kernel: LustreError: 7345:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 07:15:28 oak-gw06 kernel: LustreError: 7345:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a30ca6c0) refcount = 2 Aug 9 07:15:28 oak-gw06 kernel: LustreError: 7345:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 07:15:28 oak-gw06 kernel: LustreError: 7345:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801836c7600/0xf077f1a82d03c22e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8e88480 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 07:15:28 oak-gw06 kernel: LustreError: 7345:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 07:15:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 07:20:34 oak-gw06 kernel: LustreError: 7361:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802d4e3e540) refcount = 2 Aug 9 07:20:34 oak-gw06 kernel: LustreError: 7361:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 07:20:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 07:20:34 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 07:25:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 07:25:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 07:25:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502288442, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880197a00000/0xf077f1a82d03ef8b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8e9793a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 07:25:42 oak-gw06 kernel: LustreError: 7366:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88041bcd36c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 07:25:42 oak-gw06 kernel: LustreError: 7366:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 07:25:42 oak-gw06 kernel: LustreError: 7366:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041bcd36c0) refcount = 2 Aug 9 07:25:42 oak-gw06 kernel: LustreError: 7366:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 07:25:42 oak-gw06 kernel: LustreError: 7366:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880197a00000/0xf077f1a82d03ef8b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8e9793a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 07:25:42 oak-gw06 kernel: LustreError: 7366:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 07:25:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 07:30:48 oak-gw06 kernel: LustreError: 7378:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880114cbf3c0) refcount = 2 Aug 9 07:30:48 oak-gw06 kernel: LustreError: 7378:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 07:30:48 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 07:30:48 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 07:35:54 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 07:35:54 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 07:35:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502289054, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802dc9b3400/0xf077f1a82d0413c6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8ea6c7a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 07:35:54 oak-gw06 kernel: LustreError: 7385:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803202cb600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 07:35:54 oak-gw06 kernel: LustreError: 7385:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 07:35:54 oak-gw06 kernel: LustreError: 7385:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803202cb600) refcount = 2 Aug 9 07:35:54 oak-gw06 kernel: LustreError: 7385:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 07:35:54 oak-gw06 kernel: LustreError: 7385:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802dc9b3400/0xf077f1a82d0413c6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8ea6c7a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 07:35:54 oak-gw06 kernel: LustreError: 7385:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 07:35:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 07:41:02 oak-gw06 kernel: LustreError: 7403:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803de390c00) refcount = 2 Aug 9 07:41:02 oak-gw06 kernel: LustreError: 7403:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 07:41:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 07:41:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 07:46:08 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 07:46:08 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 07:46:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502289668, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880195bb2e00/0xf077f1a82d0524f7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8eb6015 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 07:46:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 07:46:08 oak-gw06 kernel: LustreError: 7407:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801953790c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 07:46:08 oak-gw06 kernel: LustreError: 7407:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 07:46:08 oak-gw06 kernel: LustreError: 7407:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801953790c0) refcount = 2 Aug 9 07:46:08 oak-gw06 kernel: LustreError: 7407:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 07:46:08 oak-gw06 kernel: LustreError: 7407:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880195bb2e00/0xf077f1a82d0524f7 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8eb6015 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 07:46:08 oak-gw06 kernel: LustreError: 7407:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 07:51:14 oak-gw06 kernel: LustreError: 7419:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880279bab0c0) refcount = 2 Aug 9 07:51:14 oak-gw06 kernel: LustreError: 7419:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 07:51:14 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 07:51:14 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 07:56:20 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 07:56:20 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 07:56:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502290280, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801ce3b0c00/0xf077f1a82d0551f2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8ec537f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 07:56:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 07:56:20 oak-gw06 kernel: LustreError: 7427:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88018abc8f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 07:56:20 oak-gw06 kernel: LustreError: 7427:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 07:56:20 oak-gw06 kernel: LustreError: 7427:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88018abc8f00) refcount = 2 Aug 9 07:56:20 oak-gw06 kernel: LustreError: 7427:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 07:56:20 oak-gw06 kernel: LustreError: 7427:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801ce3b0c00/0xf077f1a82d0551f2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8ec537f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 07:56:20 oak-gw06 kernel: LustreError: 7427:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 08:01:27 oak-gw06 kernel: LustreError: 7471:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ca452f00) refcount = 2 Aug 9 08:01:27 oak-gw06 kernel: LustreError: 7471:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 08:01:27 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 08:01:27 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 08:06:37 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 08:06:37 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 08:06:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502290897, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800b5b1fc00/0xf077f1a82d057aef lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8ed4990 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 08:06:37 oak-gw06 kernel: LustreError: 7475:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800560b1180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 08:06:37 oak-gw06 kernel: LustreError: 7475:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 08:06:37 oak-gw06 kernel: LustreError: 7475:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800560b1180) refcount = 2 Aug 9 08:06:37 oak-gw06 kernel: LustreError: 7475:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 08:06:37 oak-gw06 kernel: LustreError: 7475:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800b5b1fc00/0xf077f1a82d057aef lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8ed4990 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 08:06:37 oak-gw06 kernel: LustreError: 7475:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 08:06:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 08:11:42 oak-gw06 kernel: LustreError: 7490:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88003687a780) refcount = 2 Aug 9 08:11:42 oak-gw06 kernel: LustreError: 7490:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 08:11:42 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 08:11:42 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 08:16:53 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 08:16:53 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 08:16:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502291513, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8804263d6200/0xf077f1a82d05cc25 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8ee3cf3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 08:16:53 oak-gw06 kernel: LustreError: 7504:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ef4e4e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 08:16:53 oak-gw06 kernel: LustreError: 7504:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 08:16:53 oak-gw06 kernel: LustreError: 7504:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ef4e4e40) refcount = 2 Aug 9 08:16:53 oak-gw06 kernel: LustreError: 7504:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 08:16:53 oak-gw06 kernel: LustreError: 7504:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8804263d6200/0xf077f1a82d05cc25 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8ee3cf3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 08:16:53 oak-gw06 kernel: LustreError: 7504:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 08:16:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 08:22:03 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 08:22:03 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 08:27:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 08:27:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 08:27:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502292131, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880197a02600/0xf077f1a82d060831 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8ef32fd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 08:27:11 oak-gw06 kernel: LustreError: 7522:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800198840c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 08:27:11 oak-gw06 kernel: LustreError: 7522:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800198840c0) refcount = 2 Aug 9 08:27:11 oak-gw06 kernel: LustreError: 7522:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 08:27:11 oak-gw06 kernel: LustreError: 7522:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880197a02600/0xf077f1a82d060831 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8ef32fd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 08:27:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 08:32:20 oak-gw06 kernel: LustreError: 7555:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88002237d540) refcount = 2 Aug 9 08:32:20 oak-gw06 kernel: LustreError: 7555:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 08:32:20 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 08:32:20 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 08:37:29 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 08:37:29 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 08:37:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502292749, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802cf3ba000/0xf077f1a82d075a92 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8f028cf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 08:37:29 oak-gw06 kernel: LustreError: 7564:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801400dc000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 08:37:29 oak-gw06 kernel: LustreError: 7564:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 08:37:29 oak-gw06 kernel: LustreError: 7564:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801400dc000) refcount = 2 Aug 9 08:37:29 oak-gw06 kernel: LustreError: 7564:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 08:37:29 oak-gw06 kernel: LustreError: 7564:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802cf3ba000/0xf077f1a82d075a92 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8f028cf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 08:37:29 oak-gw06 kernel: LustreError: 7564:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 08:37:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 08:38:53 oak-gw06 kernel: LustreError: 11-0: oak-OST0004-osc-ffff88041b99c000: operation ldlm_enqueue to node 10.0.2.102@o2ib5 failed: rc = -107 Aug 9 08:38:53 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 08:38:53 oak-gw06 kernel: Lustre: oak-OST0004-osc-ffff88041b99c000: Connection to oak-OST0004 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 9 08:38:54 oak-gw06 kernel: LustreError: 11-0: oak-OST0024-osc-ffff88041b99c000: operation ldlm_cancel to node 10.0.2.102@o2ib5 failed: rc = -19 Aug 9 08:38:54 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 08:38:54 oak-gw06 kernel: Lustre: oak-OST0024-osc-ffff88041b99c000: Connection to oak-OST0024 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 9 08:38:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 08:38:58 oak-gw06 kernel: LustreError: 11-0: oak-OST0014-osc-ffff88041b99c000: operation ldlm_enqueue to node 10.0.2.102@o2ib5 failed: rc = -107 Aug 9 08:38:58 oak-gw06 kernel: Lustre: oak-OST0014-osc-ffff88041b99c000: Connection to oak-OST0014 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 9 08:39:04 oak-gw06 kernel: Lustre: oak-OST000c-osc-ffff88041b99c000: Connection to oak-OST000c (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 9 08:39:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 08:39:09 oak-gw06 kernel: LustreError: 11-0: oak-OST0002-osc-ffff88041b99c000: operation obd_ping to node 10.0.2.102@o2ib5 failed: rc = -107 Aug 9 08:39:09 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Aug 9 08:39:34 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1502293174/real 1502293174] req@ffff88023921a100 x1566269172001456/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1502293180 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 9 08:39:34 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 250 previous similar messages Aug 9 08:42:22 oak-gw06 kernel: Lustre: oak-OST0022-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 9 08:42:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 08:42:39 oak-gw06 kernel: LustreError: 7627:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042d8a2840) refcount = 2 Aug 9 08:42:39 oak-gw06 kernel: LustreError: 7627:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 08:43:14 oak-gw06 kernel: Lustre: DEBUG MARKER: Wed Aug 9 08:43:14 2017 Aug 9 08:47:47 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 08:47:47 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 08:47:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502293367, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88020a7af600/0xf077f1a82d07b369 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8f11f96 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 08:47:47 oak-gw06 kernel: LustreError: 7677:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880072f40600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 08:47:47 oak-gw06 kernel: LustreError: 7677:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 08:47:47 oak-gw06 kernel: LustreError: 7677:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880072f40600) refcount = 2 Aug 9 08:47:47 oak-gw06 kernel: LustreError: 7677:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 08:47:47 oak-gw06 kernel: LustreError: 7677:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88020a7af600/0xf077f1a82d07b369 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8f11f96 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 08:47:47 oak-gw06 kernel: LustreError: 7677:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 08:47:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 08:52:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 08:52:54 oak-gw06 kernel: Lustre: Skipped 22 previous similar messages Aug 9 08:57:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 08:57:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 08:57:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502293979, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803f73dce00/0xf077f1a82d0831fc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8f21346 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 08:57:59 oak-gw06 kernel: LustreError: 7697:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801a1bad240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 08:57:59 oak-gw06 kernel: LustreError: 7697:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801a1bad240) refcount = 2 Aug 9 08:57:59 oak-gw06 kernel: LustreError: 7697:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 08:57:59 oak-gw06 kernel: LustreError: 7697:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803f73dce00/0xf077f1a82d0831fc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8f21346 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 08:57:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 09:03:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 09:03:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 09:08:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 09:08:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 09:08:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502294596, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880046be3000/0xf077f1a82d0868f9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8f30838 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 09:08:16 oak-gw06 kernel: LustreError: 7758:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801b1cf8840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 09:08:16 oak-gw06 kernel: LustreError: 7758:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801b1cf8840) refcount = 2 Aug 9 09:08:16 oak-gw06 kernel: LustreError: 7758:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 09:08:16 oak-gw06 kernel: LustreError: 7758:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880046be3000/0xf077f1a82d0868f9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8f30838 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 09:08:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 09:13:24 oak-gw06 kernel: LustreError: 7768:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801b1cf8840) refcount = 2 Aug 9 09:13:24 oak-gw06 kernel: LustreError: 7768:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 09:13:24 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 09:13:24 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 09:18:32 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 09:18:32 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 09:18:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502295212, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801dbb63800/0xf077f1a82d08ae74 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8f3fd15 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 09:18:32 oak-gw06 kernel: LustreError: 7776:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801eef73540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 09:18:32 oak-gw06 kernel: LustreError: 7776:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 09:18:32 oak-gw06 kernel: LustreError: 7776:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801eef73540) refcount = 2 Aug 9 09:18:32 oak-gw06 kernel: LustreError: 7776:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 09:18:32 oak-gw06 kernel: LustreError: 7776:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801dbb63800/0xf077f1a82d08ae74 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8f3fd15 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 09:18:32 oak-gw06 kernel: LustreError: 7776:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 09:18:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 09:23:43 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 09:23:43 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 09:28:53 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 09:28:53 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 09:28:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502295833, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801bdf93600/0xf077f1a82d08f5bd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8f4f1f2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 09:28:53 oak-gw06 kernel: LustreError: 7790:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88041b30a900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 09:28:53 oak-gw06 kernel: LustreError: 7790:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041b30a900) refcount = 2 Aug 9 09:28:53 oak-gw06 kernel: LustreError: 7790:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 09:28:53 oak-gw06 kernel: LustreError: 7790:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801bdf93600/0xf077f1a82d08f5bd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8f4f1f2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 09:28:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 09:34:04 oak-gw06 kernel: LustreError: 7806:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880019884cc0) refcount = 2 Aug 9 09:34:04 oak-gw06 kernel: LustreError: 7806:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 09:34:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 09:34:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 09:39:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 09:39:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 09:39:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502296451, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88016ff48600/0xf077f1a82d092be8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8f5e71c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 09:39:11 oak-gw06 kernel: LustreError: 7809:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88002237de40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 09:39:11 oak-gw06 kernel: LustreError: 7809:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 09:39:11 oak-gw06 kernel: LustreError: 7809:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88002237de40) refcount = 2 Aug 9 09:39:11 oak-gw06 kernel: LustreError: 7809:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 09:39:11 oak-gw06 kernel: LustreError: 7809:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88016ff48600/0xf077f1a82d092be8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8f5e71c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 09:39:11 oak-gw06 kernel: LustreError: 7809:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 09:39:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 09:44:19 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 09:44:19 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 09:49:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 09:49:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 09:49:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502297068, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88038dad4800/0xf077f1a82d096e99 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8f6ddc0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 09:49:28 oak-gw06 kernel: LustreError: 7828:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880287fc7900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 09:49:28 oak-gw06 kernel: LustreError: 7828:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880287fc7900) refcount = 2 Aug 9 09:49:28 oak-gw06 kernel: LustreError: 7828:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 09:49:28 oak-gw06 kernel: LustreError: 7828:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88038dad4800/0xf077f1a82d096e99 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8f6ddc0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 09:49:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 09:54:34 oak-gw06 kernel: LustreError: 7844:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880417bf2240) refcount = 2 Aug 9 09:54:34 oak-gw06 kernel: LustreError: 7844:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 09:54:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 09:54:34 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 09:59:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 09:59:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 09:59:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502297684, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880274aac600/0xf077f1a82d09de45 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8f7d330 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 09:59:44 oak-gw06 kernel: LustreError: 7858:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88022d4f1180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 09:59:44 oak-gw06 kernel: LustreError: 7858:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 09:59:44 oak-gw06 kernel: LustreError: 7858:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022d4f1180) refcount = 2 Aug 9 09:59:44 oak-gw06 kernel: LustreError: 7858:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 09:59:44 oak-gw06 kernel: LustreError: 7858:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880274aac600/0xf077f1a82d09de45 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8f7d330 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 09:59:44 oak-gw06 kernel: LustreError: 7858:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 09:59:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 10:04:55 oak-gw06 kernel: LustreError: 7904:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801853dd000) refcount = 2 Aug 9 10:04:55 oak-gw06 kernel: LustreError: 7904:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 10:04:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 10:04:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 10:10:03 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 10:10:03 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 10:10:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502298303, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880357007800/0xf077f1a82d0a1733 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8f8c7ea expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 10:10:03 oak-gw06 kernel: LustreError: 7919:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803d3ae6540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 10:10:03 oak-gw06 kernel: LustreError: 7919:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 10:10:03 oak-gw06 kernel: LustreError: 7919:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3ae6540) refcount = 2 Aug 9 10:10:03 oak-gw06 kernel: LustreError: 7919:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 10:10:03 oak-gw06 kernel: LustreError: 7919:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880357007800/0xf077f1a82d0a1733 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8f8c7ea expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 10:10:03 oak-gw06 kernel: LustreError: 7919:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 10:10:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 10:15:12 oak-gw06 kernel: LustreError: 7924:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802136c6480) refcount = 2 Aug 9 10:15:12 oak-gw06 kernel: LustreError: 7924:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 10:15:12 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 10:15:12 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 10:20:19 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 10:20:19 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 10:20:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502298919, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88042e7c9400/0xf077f1a82d0a60eb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8f9bd3e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 10:20:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 10:20:19 oak-gw06 kernel: LustreError: 7939:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88022a2d46c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 10:20:19 oak-gw06 kernel: LustreError: 7939:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 10:20:19 oak-gw06 kernel: LustreError: 7939:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022a2d46c0) refcount = 2 Aug 9 10:20:19 oak-gw06 kernel: LustreError: 7939:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 10:20:19 oak-gw06 kernel: LustreError: 7939:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88042e7c9400/0xf077f1a82d0a60eb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8f9bd3e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 10:20:19 oak-gw06 kernel: LustreError: 7939:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 10:25:27 oak-gw06 kernel: LustreError: 7945:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801220e8780) refcount = 2 Aug 9 10:25:27 oak-gw06 kernel: LustreError: 7945:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 10:25:27 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 10:25:27 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 10:30:32 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 10:30:32 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 10:30:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502299532, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802ebc21600/0xf077f1a82d0aa501 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8faafd6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 10:30:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 10:30:32 oak-gw06 kernel: LustreError: 7957:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802efa52cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 10:30:32 oak-gw06 kernel: LustreError: 7957:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 10:30:32 oak-gw06 kernel: LustreError: 7957:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802efa52cc0) refcount = 2 Aug 9 10:30:32 oak-gw06 kernel: LustreError: 7957:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 10:30:32 oak-gw06 kernel: LustreError: 7957:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802ebc21600/0xf077f1a82d0aa501 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8faafd6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 10:30:32 oak-gw06 kernel: LustreError: 7957:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 10:35:39 oak-gw06 kernel: LustreError: 7961:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802099c6540) refcount = 2 Aug 9 10:35:39 oak-gw06 kernel: LustreError: 7961:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 10:35:39 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 10:35:39 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 10:40:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 10:40:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 10:40:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502300149, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88005c27aa00/0xf077f1a82d0ac79f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8fba49e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 10:40:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 10:40:49 oak-gw06 kernel: LustreError: 7975:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88015fdb3780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 10:40:49 oak-gw06 kernel: LustreError: 7975:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 10:40:49 oak-gw06 kernel: LustreError: 7975:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88015fdb3780) refcount = 2 Aug 9 10:40:49 oak-gw06 kernel: LustreError: 7975:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 10:40:49 oak-gw06 kernel: LustreError: 7975:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88005c27aa00/0xf077f1a82d0ac79f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8fba49e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 10:40:49 oak-gw06 kernel: LustreError: 7975:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 10:45:59 oak-gw06 kernel: LustreError: 7979:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801b7c78cc0) refcount = 2 Aug 9 10:45:59 oak-gw06 kernel: LustreError: 7979:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 10:45:59 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 10:45:59 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 10:51:06 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 10:51:06 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 10:51:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502300766, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88028ef5e000/0xf077f1a82d0b11c0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8fc99ba expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 10:51:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 10:51:06 oak-gw06 kernel: LustreError: 8023:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880340754300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 10:51:06 oak-gw06 kernel: LustreError: 8023:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 10:51:06 oak-gw06 kernel: LustreError: 8023:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880340754300) refcount = 2 Aug 9 10:51:06 oak-gw06 kernel: LustreError: 8023:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 10:51:06 oak-gw06 kernel: LustreError: 8023:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88028ef5e000/0xf077f1a82d0b11c0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8fc99ba expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 10:51:06 oak-gw06 kernel: LustreError: 8023:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 10:56:16 oak-gw06 kernel: LustreError: 8216:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8802eea9f480) refcount = 2 Aug 9 10:56:16 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 10:56:16 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 11:01:22 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 11:01:22 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 11:01:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502301382, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802dea39600/0xf077f1a82d200d6e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8fd8de1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 11:01:22 oak-gw06 kernel: LustreError: 8335:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880190b13480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 11:01:22 oak-gw06 kernel: LustreError: 8335:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 11:01:22 oak-gw06 kernel: LustreError: 8335:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880190b13480) refcount = 2 Aug 9 11:01:22 oak-gw06 kernel: LustreError: 8335:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 11:01:22 oak-gw06 kernel: LustreError: 8335:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802dea39600/0xf077f1a82d200d6e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8fd8de1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 11:01:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 11:06:31 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 11:06:31 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 11:11:38 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 11:11:38 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 11:11:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502301998, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88033aee4e00/0xf077f1a82d2f5be6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8fe82f6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 11:11:38 oak-gw06 kernel: LustreError: 8461:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803b2f626c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 11:11:38 oak-gw06 kernel: LustreError: 8461:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803b2f626c0) refcount = 2 Aug 9 11:11:38 oak-gw06 kernel: LustreError: 8461:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 11:11:38 oak-gw06 kernel: LustreError: 8461:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88033aee4e00/0xf077f1a82d2f5be6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8fe82f6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 11:11:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 11:16:46 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 11:16:46 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 11:21:55 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 11:21:55 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 11:21:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502302615, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8804040bc800/0xf077f1a82d30a51e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb8ff7882 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 11:21:55 oak-gw06 kernel: LustreError: 8474:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880161439900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 11:21:55 oak-gw06 kernel: LustreError: 8474:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880161439900) refcount = 2 Aug 9 11:21:55 oak-gw06 kernel: LustreError: 8474:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 11:21:55 oak-gw06 kernel: LustreError: 8474:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8804040bc800/0xf077f1a82d30a51e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb8ff7882 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 11:21:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 11:27:02 oak-gw06 kernel: LustreError: 8481:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88020e1b2000) refcount = 2 Aug 9 11:27:02 oak-gw06 kernel: LustreError: 8481:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 11:27:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 11:27:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 11:32:12 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 11:32:12 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 11:32:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502303232, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803bd4f9c00/0xf077f1a82d30e2f8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9006eb6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 11:32:12 oak-gw06 kernel: LustreError: 8493:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88009fcc5480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 11:32:12 oak-gw06 kernel: LustreError: 8493:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 11:32:12 oak-gw06 kernel: LustreError: 8493:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88009fcc5480) refcount = 2 Aug 9 11:32:12 oak-gw06 kernel: LustreError: 8493:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 11:32:12 oak-gw06 kernel: LustreError: 8493:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803bd4f9c00/0xf077f1a82d30e2f8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9006eb6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 11:32:12 oak-gw06 kernel: LustreError: 8493:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 11:32:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 11:37:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 11:37:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 11:42:30 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 11:42:30 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 11:42:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502303850, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803bf369a00/0xf077f1a82d31008e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb901649d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 11:42:30 oak-gw06 kernel: LustreError: 8506:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801081583c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 11:42:30 oak-gw06 kernel: LustreError: 8506:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801081583c0) refcount = 2 Aug 9 11:42:30 oak-gw06 kernel: LustreError: 8506:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 11:42:30 oak-gw06 kernel: LustreError: 8506:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803bf369a00/0xf077f1a82d31008e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb901649d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 11:42:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 11:47:39 oak-gw06 kernel: LustreError: 8510:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880372617cc0) refcount = 2 Aug 9 11:47:39 oak-gw06 kernel: LustreError: 8510:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 11:47:39 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 11:47:39 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 11:52:45 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 11:52:45 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 11:52:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502304465, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880112027600/0xf077f1a82d311ea2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb90258c4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 11:52:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 11:52:45 oak-gw06 kernel: LustreError: 8525:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880372617300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 11:52:45 oak-gw06 kernel: LustreError: 8525:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 11:52:45 oak-gw06 kernel: LustreError: 8525:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880372617300) refcount = 2 Aug 9 11:52:45 oak-gw06 kernel: LustreError: 8525:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 11:52:45 oak-gw06 kernel: LustreError: 8525:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880112027600/0xf077f1a82d311ea2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb90258c4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 11:52:45 oak-gw06 kernel: LustreError: 8525:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 11:57:54 oak-gw06 kernel: LustreError: 8530:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802dcaa9600) refcount = 2 Aug 9 11:57:54 oak-gw06 kernel: LustreError: 8530:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 11:57:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 11:57:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 12:03:01 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 12:03:01 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 12:03:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502305081, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800127ee600/0xf077f1a82d3168d8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9034e6c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 12:03:01 oak-gw06 kernel: LustreError: 8578:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88022e65bcc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 12:03:01 oak-gw06 kernel: LustreError: 8578:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 12:03:01 oak-gw06 kernel: LustreError: 8578:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022e65bcc0) refcount = 2 Aug 9 12:03:01 oak-gw06 kernel: LustreError: 8578:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 12:03:01 oak-gw06 kernel: LustreError: 8578:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800127ee600/0xf077f1a82d3168d8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9034e6c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 12:03:01 oak-gw06 kernel: LustreError: 8578:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 12:03:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 12:08:09 oak-gw06 kernel: LustreError: 8582:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880115158780) refcount = 2 Aug 9 12:08:09 oak-gw06 kernel: LustreError: 8582:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 12:08:09 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 12:08:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 12:13:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 12:13:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 12:13:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502305696, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88018070e600/0xf077f1a82d31bec2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb904437a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 12:13:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 12:13:16 oak-gw06 kernel: LustreError: 8598:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88030d839000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 12:13:16 oak-gw06 kernel: LustreError: 8598:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 12:13:16 oak-gw06 kernel: LustreError: 8598:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88030d839000) refcount = 2 Aug 9 12:13:16 oak-gw06 kernel: LustreError: 8598:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 12:13:16 oak-gw06 kernel: LustreError: 8598:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88018070e600/0xf077f1a82d31bec2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb904437a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 12:13:16 oak-gw06 kernel: LustreError: 8598:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 12:18:21 oak-gw06 kernel: LustreError: 8601:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880041f056c0) refcount = 2 Aug 9 12:18:21 oak-gw06 kernel: LustreError: 8601:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 12:18:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 12:18:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 12:23:27 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 12:23:27 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 12:23:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502306307, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88000a78da00/0xf077f1a82d31f422 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9053643 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 12:23:27 oak-gw06 kernel: LustreError: 8613:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880041f05f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 12:23:27 oak-gw06 kernel: LustreError: 8613:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 12:23:27 oak-gw06 kernel: LustreError: 8613:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880041f05f00) refcount = 2 Aug 9 12:23:27 oak-gw06 kernel: LustreError: 8613:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 12:23:27 oak-gw06 kernel: LustreError: 8613:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88000a78da00/0xf077f1a82d31f422 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9053643 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 12:23:27 oak-gw06 kernel: LustreError: 8613:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 12:23:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 12:28:34 oak-gw06 kernel: LustreError: 8622:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801872d4e40) refcount = 2 Aug 9 12:28:34 oak-gw06 kernel: LustreError: 8622:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 12:28:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 12:28:34 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 12:33:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 12:33:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 12:33:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502306924, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800a7224600/0xf077f1a82d3237e4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9062c9a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 12:33:44 oak-gw06 kernel: LustreError: 8633:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880040adb480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 12:33:44 oak-gw06 kernel: LustreError: 8633:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 12:33:44 oak-gw06 kernel: LustreError: 8633:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880040adb480) refcount = 2 Aug 9 12:33:44 oak-gw06 kernel: LustreError: 8633:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 12:33:44 oak-gw06 kernel: LustreError: 8633:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800a7224600/0xf077f1a82d3237e4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9062c9a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 12:33:44 oak-gw06 kernel: LustreError: 8633:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 12:33:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 12:38:53 oak-gw06 kernel: LustreError: 8637:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802f0b739c0) refcount = 2 Aug 9 12:38:53 oak-gw06 kernel: LustreError: 8637:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 12:38:53 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 12:38:53 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 12:44:04 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 12:44:04 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 12:44:04 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502307544, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88010a994200/0xf077f1a82d326f6d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb90722dc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 12:44:04 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 12:44:04 oak-gw06 kernel: LustreError: 8648:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880049e37900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 12:44:04 oak-gw06 kernel: LustreError: 8648:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 12:44:04 oak-gw06 kernel: LustreError: 8648:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880049e37900) refcount = 2 Aug 9 12:44:04 oak-gw06 kernel: LustreError: 8648:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 12:44:04 oak-gw06 kernel: LustreError: 8648:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88010a994200/0xf077f1a82d326f6d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb90722dc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 12:44:04 oak-gw06 kernel: LustreError: 8648:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 12:49:11 oak-gw06 kernel: LustreError: 8657:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880040adb540) refcount = 2 Aug 9 12:49:11 oak-gw06 kernel: LustreError: 8657:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 12:49:11 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 12:49:11 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 12:54:17 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 12:54:17 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 12:54:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502308157, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800a7227200/0xf077f1a82d3291fd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9081734 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 12:54:17 oak-gw06 kernel: LustreError: 8669:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88007c960540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 12:54:17 oak-gw06 kernel: LustreError: 8669:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 12:54:17 oak-gw06 kernel: LustreError: 8669:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007c960540) refcount = 2 Aug 9 12:54:17 oak-gw06 kernel: LustreError: 8669:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 12:54:17 oak-gw06 kernel: LustreError: 8669:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800a7227200/0xf077f1a82d3291fd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9081734 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 12:54:17 oak-gw06 kernel: LustreError: 8669:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 12:54:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 12:59:22 oak-gw06 kernel: LustreError: 8682:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88016c7da480) refcount = 2 Aug 9 12:59:22 oak-gw06 kernel: LustreError: 8682:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 12:59:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 12:59:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 13:04:32 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 13:04:32 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 13:04:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502308772, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88022e5fe400/0xf077f1a82d32fb74 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9090b8c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 13:04:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 13:04:32 oak-gw06 kernel: LustreError: 8725:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88008ddc4d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 13:04:32 oak-gw06 kernel: LustreError: 8725:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 13:04:32 oak-gw06 kernel: LustreError: 8725:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88008ddc4d80) refcount = 2 Aug 9 13:04:32 oak-gw06 kernel: LustreError: 8725:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 13:04:32 oak-gw06 kernel: LustreError: 8725:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88022e5fe400/0xf077f1a82d32fb74 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9090b8c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 13:04:32 oak-gw06 kernel: LustreError: 8725:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 13:09:38 oak-gw06 kernel: LustreError: 8734:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041d96f840) refcount = 2 Aug 9 13:09:38 oak-gw06 kernel: LustreError: 8734:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 13:09:38 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 13:09:38 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 13:14:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 13:14:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 13:14:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502309384, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88016278c400/0xf077f1a82d333025 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb909ff04 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 13:14:44 oak-gw06 kernel: LustreError: 8746:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880003c87600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 13:14:44 oak-gw06 kernel: LustreError: 8746:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 13:14:44 oak-gw06 kernel: LustreError: 8746:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880003c87600) refcount = 2 Aug 9 13:14:44 oak-gw06 kernel: LustreError: 8746:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 13:14:44 oak-gw06 kernel: LustreError: 8746:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88016278c400/0xf077f1a82d333025 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb909ff04 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 13:14:44 oak-gw06 kernel: LustreError: 8746:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 13:14:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 13:19:53 oak-gw06 kernel: LustreError: 8750:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880003c87e40) refcount = 2 Aug 9 13:19:53 oak-gw06 kernel: LustreError: 8750:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 13:19:53 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 13:19:53 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 13:24:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 13:24:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 13:24:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502309999, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880398a9b800/0xf077f1a82d33554e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb90af2c2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 13:24:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 13:24:59 oak-gw06 kernel: LustreError: 8776:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880416e6fa80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 13:24:59 oak-gw06 kernel: LustreError: 8776:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 13:24:59 oak-gw06 kernel: LustreError: 8776:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880416e6fa80) refcount = 2 Aug 9 13:24:59 oak-gw06 kernel: LustreError: 8776:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 13:24:59 oak-gw06 kernel: LustreError: 8776:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880398a9b800/0xf077f1a82d33554e lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb90af2c2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 13:24:59 oak-gw06 kernel: LustreError: 8776:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 13:30:06 oak-gw06 kernel: LustreError: 8787:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880115194900) refcount = 2 Aug 9 13:30:06 oak-gw06 kernel: LustreError: 8787:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 13:30:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 13:30:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 13:35:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 13:35:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 13:35:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502310616, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802061ebe00/0xf077f1a82d33b21c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb90be7d7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 13:35:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 13:35:16 oak-gw06 kernel: LustreError: 8795:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801627bae40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 13:35:16 oak-gw06 kernel: LustreError: 8795:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 13:35:16 oak-gw06 kernel: LustreError: 8795:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801627bae40) refcount = 2 Aug 9 13:35:16 oak-gw06 kernel: LustreError: 8795:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 13:35:16 oak-gw06 kernel: LustreError: 8795:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802061ebe00/0xf077f1a82d33b21c lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb90be7d7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 13:35:16 oak-gw06 kernel: LustreError: 8795:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 13:40:22 oak-gw06 kernel: LustreError: 8808:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880175f09000) refcount = 2 Aug 9 13:40:22 oak-gw06 kernel: LustreError: 8808:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 13:40:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 13:40:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 13:45:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 13:45:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 13:45:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502311228, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801f76c6200/0xf077f1a82d33fe04 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb90cdc13 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 13:45:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 13:45:28 oak-gw06 kernel: LustreError: 8813:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88029d26d000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 13:45:28 oak-gw06 kernel: LustreError: 8813:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 13:45:28 oak-gw06 kernel: LustreError: 8813:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88029d26d000) refcount = 2 Aug 9 13:45:28 oak-gw06 kernel: LustreError: 8813:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 13:45:28 oak-gw06 kernel: LustreError: 8813:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801f76c6200/0xf077f1a82d33fe04 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb90cdc13 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 13:45:28 oak-gw06 kernel: LustreError: 8813:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 13:50:37 oak-gw06 kernel: LustreError: 8824:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801cb7cd240) refcount = 2 Aug 9 13:50:37 oak-gw06 kernel: LustreError: 8824:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 13:50:37 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 13:50:37 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 13:55:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 13:55:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 13:55:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502311844, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88012e559800/0xf077f1a82d342caa lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb90dd160 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 13:55:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 13:55:44 oak-gw06 kernel: LustreError: 8832:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88013f28f000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 13:55:44 oak-gw06 kernel: LustreError: 8832:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 13:55:44 oak-gw06 kernel: LustreError: 8832:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88013f28f000) refcount = 2 Aug 9 13:55:44 oak-gw06 kernel: LustreError: 8832:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 13:55:44 oak-gw06 kernel: LustreError: 8832:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88012e559800/0xf077f1a82d342caa lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb90dd160 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 13:55:44 oak-gw06 kernel: LustreError: 8832:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 14:00:54 oak-gw06 kernel: LustreError: 8844:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880145bbb900) refcount = 2 Aug 9 14:00:54 oak-gw06 kernel: LustreError: 8844:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 14:00:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 14:00:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 14:06:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 14:06:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 14:06:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502312460, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802c93e1800/0xf077f1a82d347c97 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb90ec6d0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 14:06:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 14:06:00 oak-gw06 kernel: LustreError: 8883:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801a1bad180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 14:06:00 oak-gw06 kernel: LustreError: 8883:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 14:06:00 oak-gw06 kernel: LustreError: 8883:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801a1bad180) refcount = 2 Aug 9 14:06:00 oak-gw06 kernel: LustreError: 8883:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 14:06:00 oak-gw06 kernel: LustreError: 8883:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802c93e1800/0xf077f1a82d347c97 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb90ec6d0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 14:06:00 oak-gw06 kernel: LustreError: 8883:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 14:11:09 oak-gw06 kernel: LustreError: 8893:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880051f19a80) refcount = 2 Aug 9 14:11:09 oak-gw06 kernel: LustreError: 8893:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 14:11:09 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 14:11:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 14:16:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 14:16:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 14:16:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502313076, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880150017600/0xf077f1a82d34a4de lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb90fbb59 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 14:16:16 oak-gw06 kernel: LustreError: 8902:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803072aa540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 14:16:16 oak-gw06 kernel: LustreError: 8902:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 14:16:16 oak-gw06 kernel: LustreError: 8902:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803072aa540) refcount = 2 Aug 9 14:16:16 oak-gw06 kernel: LustreError: 8902:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 14:16:16 oak-gw06 kernel: LustreError: 8902:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880150017600/0xf077f1a82d34a4de lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb90fbb59 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 14:16:16 oak-gw06 kernel: LustreError: 8902:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 14:16:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 14:21:26 oak-gw06 kernel: LustreError: 8917:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880128f15300) refcount = 2 Aug 9 14:21:26 oak-gw06 kernel: LustreError: 8917:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 14:21:26 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 14:21:26 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 14:26:33 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 14:26:33 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 14:26:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502313693, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800555c4800/0xf077f1a82d3565c0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb910b132 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 14:26:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 14:26:33 oak-gw06 kernel: LustreError: 8945:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801b9e693c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 14:26:33 oak-gw06 kernel: LustreError: 8945:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 14:26:33 oak-gw06 kernel: LustreError: 8945:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801b9e693c0) refcount = 2 Aug 9 14:26:33 oak-gw06 kernel: LustreError: 8945:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 14:26:33 oak-gw06 kernel: LustreError: 8945:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800555c4800/0xf077f1a82d3565c0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb910b132 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 14:26:33 oak-gw06 kernel: LustreError: 8945:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 14:31:42 oak-gw06 kernel: LustreError: 8960:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803b89ded80) refcount = 2 Aug 9 14:31:42 oak-gw06 kernel: LustreError: 8960:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 14:31:42 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 14:31:42 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 14:36:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 14:36:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 14:36:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502314309, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801cdfd1c00/0xf077f1a82d37fd1c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb911a6b0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 14:36:49 oak-gw06 kernel: LustreError: 8968:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a73d5240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 14:36:49 oak-gw06 kernel: LustreError: 8968:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 14:36:49 oak-gw06 kernel: LustreError: 8968:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a73d5240) refcount = 2 Aug 9 14:36:49 oak-gw06 kernel: LustreError: 8968:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 14:36:49 oak-gw06 kernel: LustreError: 8968:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801cdfd1c00/0xf077f1a82d37fd1c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb911a6b0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 14:36:49 oak-gw06 kernel: LustreError: 8968:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 14:36:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 14:41:58 oak-gw06 kernel: LustreError: 9007:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800856a5cc0) refcount = 2 Aug 9 14:41:58 oak-gw06 kernel: LustreError: 9007:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 14:41:58 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 14:41:58 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 14:47:05 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 14:47:05 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 14:47:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502314925, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803f034d800/0xf077f1a82d3afecf lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9129c5f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 14:47:05 oak-gw06 kernel: LustreError: 9200:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802527bd600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 14:47:05 oak-gw06 kernel: LustreError: 9200:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 14:47:05 oak-gw06 kernel: LustreError: 9200:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802527bd600) refcount = 2 Aug 9 14:47:05 oak-gw06 kernel: LustreError: 9200:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 14:47:05 oak-gw06 kernel: LustreError: 9200:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803f034d800/0xf077f1a82d3afecf lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9129c5f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 14:47:05 oak-gw06 kernel: LustreError: 9200:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 14:47:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 14:52:12 oak-gw06 kernel: LustreError: 9384:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880125359cc0) refcount = 2 Aug 9 14:52:12 oak-gw06 kernel: LustreError: 9384:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 14:52:12 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 14:52:12 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 14:57:18 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 14:57:18 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 14:57:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502315538, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802a53ddc00/0xf077f1a82d5d5f03 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9139071 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 14:57:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 14:57:18 oak-gw06 kernel: LustreError: 9394:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88012fb69780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 14:57:18 oak-gw06 kernel: LustreError: 9394:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 14:57:18 oak-gw06 kernel: LustreError: 9394:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88012fb69780) refcount = 2 Aug 9 14:57:18 oak-gw06 kernel: LustreError: 9394:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 14:57:18 oak-gw06 kernel: LustreError: 9394:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802a53ddc00/0xf077f1a82d5d5f03 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9139071 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 14:57:18 oak-gw06 kernel: LustreError: 9394:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 15:02:28 oak-gw06 kernel: LustreError: 9443:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880272d1ab40) refcount = 2 Aug 9 15:02:28 oak-gw06 kernel: LustreError: 9443:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 15:02:28 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 15:02:28 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 15:07:37 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 15:07:37 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 15:07:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502316157, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880230639600/0xf077f1a82d5e3413 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9148586 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 15:07:37 oak-gw06 kernel: LustreError: 9450:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801c8969780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 15:07:37 oak-gw06 kernel: LustreError: 9450:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 15:07:37 oak-gw06 kernel: LustreError: 9450:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c8969780) refcount = 2 Aug 9 15:07:37 oak-gw06 kernel: LustreError: 9450:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 15:07:37 oak-gw06 kernel: LustreError: 9450:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880230639600/0xf077f1a82d5e3413 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9148586 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 15:07:37 oak-gw06 kernel: LustreError: 9450:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 15:07:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 15:12:44 oak-gw06 kernel: LustreError: 9467:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022b70a600) refcount = 1 Aug 9 15:12:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 15:12:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 15:17:52 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 15:17:52 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 15:17:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502316772, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880345a15200/0xf077f1a82d5ebffe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9157b35 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 15:17:52 oak-gw06 kernel: LustreError: 9475:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801fe3756c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 15:17:52 oak-gw06 kernel: LustreError: 9475:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 15:17:52 oak-gw06 kernel: LustreError: 9475:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801fe3756c0) refcount = 2 Aug 9 15:17:52 oak-gw06 kernel: LustreError: 9475:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 15:17:52 oak-gw06 kernel: LustreError: 9475:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880345a15200/0xf077f1a82d5ebffe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9157b35 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 15:17:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 15:22:59 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 15:22:59 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 15:28:07 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 15:28:07 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 15:28:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502317387, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015a661000/0xf077f1a82d6068ce lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb91671bd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 15:28:07 oak-gw06 kernel: LustreError: 9516:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88030ef77240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 15:28:07 oak-gw06 kernel: LustreError: 9516:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88030ef77240) refcount = 2 Aug 9 15:28:07 oak-gw06 kernel: LustreError: 9516:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 15:28:07 oak-gw06 kernel: LustreError: 9516:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015a661000/0xf077f1a82d6068ce lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb91671bd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 15:28:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 15:33:16 oak-gw06 kernel: LustreError: 9526:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff88021179be40) refcount = 2 Aug 9 15:33:16 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 15:33:16 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 15:38:25 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 15:38:25 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 15:38:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502318005, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802975f7a00/0xf077f1a82d61d80f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9176781 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 15:38:25 oak-gw06 kernel: LustreError: 9528:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88021179b180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 15:38:25 oak-gw06 kernel: LustreError: 9528:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 15:38:25 oak-gw06 kernel: LustreError: 9528:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88021179b180) refcount = 2 Aug 9 15:38:25 oak-gw06 kernel: LustreError: 9528:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 15:38:25 oak-gw06 kernel: LustreError: 9528:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802975f7a00/0xf077f1a82d61d80f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9176781 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 15:38:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 15:43:32 oak-gw06 kernel: LustreError: 9543:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88021179b180) refcount = 2 Aug 9 15:43:32 oak-gw06 kernel: LustreError: 9543:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 15:43:32 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 15:43:32 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 15:48:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 15:48:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 15:48:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502318622, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802975f7a00/0xf077f1a82d620095 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9185ca4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 15:48:42 oak-gw06 kernel: LustreError: 9547:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880021775300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 15:48:42 oak-gw06 kernel: LustreError: 9547:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 15:48:42 oak-gw06 kernel: LustreError: 9547:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880021775300) refcount = 2 Aug 9 15:48:42 oak-gw06 kernel: LustreError: 9547:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 15:48:42 oak-gw06 kernel: LustreError: 9547:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802975f7a00/0xf077f1a82d620095 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9185ca4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 15:48:42 oak-gw06 kernel: LustreError: 9547:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 15:48:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 15:53:48 oak-gw06 kernel: LustreError: 9562:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801e53ae540) refcount = 2 Aug 9 15:53:48 oak-gw06 kernel: LustreError: 9562:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 15:53:48 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 15:53:48 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 15:58:55 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 15:58:55 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 15:58:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502319235, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880152e78a00/0xf077f1a82d627573 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb919500e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 15:58:55 oak-gw06 kernel: LustreError: 9566:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880269bdfc00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 15:58:55 oak-gw06 kernel: LustreError: 9566:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 15:58:55 oak-gw06 kernel: LustreError: 9566:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880269bdfc00) refcount = 2 Aug 9 15:58:55 oak-gw06 kernel: LustreError: 9566:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 15:58:55 oak-gw06 kernel: LustreError: 9566:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880152e78a00/0xf077f1a82d627573 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb919500e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 15:58:55 oak-gw06 kernel: LustreError: 9566:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 15:58:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 16:04:01 oak-gw06 kernel: LustreError: 9625:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802667ea9c0) refcount = 2 Aug 9 16:04:01 oak-gw06 kernel: LustreError: 9625:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 16:04:01 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 16:04:01 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 16:09:12 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 16:09:12 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 16:09:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502319851, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880136cf3000/0xf077f1a82d62df99 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb91a453f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 16:09:12 oak-gw06 kernel: LustreError: 9629:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801142423c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 16:09:12 oak-gw06 kernel: LustreError: 9629:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 16:09:12 oak-gw06 kernel: LustreError: 9629:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801142423c0) refcount = 2 Aug 9 16:09:12 oak-gw06 kernel: LustreError: 9629:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 16:09:12 oak-gw06 kernel: LustreError: 9629:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880136cf3000/0xf077f1a82d62df99 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb91a453f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 16:09:12 oak-gw06 kernel: LustreError: 9629:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 16:09:12 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 16:14:21 oak-gw06 kernel: LustreError: 9644:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88035f9dfd80) refcount = 2 Aug 9 16:14:21 oak-gw06 kernel: LustreError: 9644:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 16:14:21 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 16:14:21 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 16:19:29 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 16:19:29 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 16:19:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502320469, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88014252b200/0xf077f1a82d6315e7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb91b3958 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 16:19:29 oak-gw06 kernel: LustreError: 9649:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801142419c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 16:19:29 oak-gw06 kernel: LustreError: 9649:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 16:19:29 oak-gw06 kernel: LustreError: 9649:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801142419c0) refcount = 2 Aug 9 16:19:29 oak-gw06 kernel: LustreError: 9649:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 16:19:29 oak-gw06 kernel: LustreError: 9649:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88014252b200/0xf077f1a82d6315e7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb91b3958 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 16:19:29 oak-gw06 kernel: LustreError: 9649:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 16:19:29 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 16:24:36 oak-gw06 kernel: LustreError: 9665:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880273809d80) refcount = 2 Aug 9 16:24:36 oak-gw06 kernel: LustreError: 9665:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 16:24:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 16:24:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 16:29:46 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 16:29:46 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 16:29:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502321086, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801b3b5bc00/0xf077f1a82d638ab0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb91c2def expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 16:29:46 oak-gw06 kernel: LustreError: 9669:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88022d4f10c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 16:29:46 oak-gw06 kernel: LustreError: 9669:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 16:29:46 oak-gw06 kernel: LustreError: 9669:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022d4f10c0) refcount = 2 Aug 9 16:29:46 oak-gw06 kernel: LustreError: 9669:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 16:29:46 oak-gw06 kernel: LustreError: 9669:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801b3b5bc00/0xf077f1a82d638ab0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb91c2def expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 16:29:46 oak-gw06 kernel: LustreError: 9669:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 16:29:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 16:34:52 oak-gw06 kernel: LustreError: 9687:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022d4f1540) refcount = 2 Aug 9 16:34:52 oak-gw06 kernel: LustreError: 9687:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 16:34:52 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 16:34:52 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 16:40:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 16:40:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 16:40:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502321700, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801ff577400/0xf077f1a82d644819 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb91d214b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 16:40:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 16:40:00 oak-gw06 kernel: LustreError: 9692:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801409ea900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 16:40:00 oak-gw06 kernel: LustreError: 9692:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 16:40:00 oak-gw06 kernel: LustreError: 9692:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801409ea900) refcount = 2 Aug 9 16:40:00 oak-gw06 kernel: LustreError: 9692:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 16:40:00 oak-gw06 kernel: LustreError: 9692:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801ff577400/0xf077f1a82d644819 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb91d214b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 16:40:00 oak-gw06 kernel: LustreError: 9692:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 16:45:08 oak-gw06 kernel: LustreError: 9702:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e46d6e40) refcount = 2 Aug 9 16:45:08 oak-gw06 kernel: LustreError: 9702:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 16:45:08 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 16:45:08 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 16:50:18 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 16:50:18 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 16:50:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502322318, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802975f7c00/0xf077f1a82d64763a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb91e15cd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 16:50:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 16:50:18 oak-gw06 kernel: LustreError: 9717:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800132a5b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 16:50:18 oak-gw06 kernel: LustreError: 9717:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 16:50:18 oak-gw06 kernel: LustreError: 9717:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800132a5b40) refcount = 2 Aug 9 16:50:18 oak-gw06 kernel: LustreError: 9717:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 16:50:18 oak-gw06 kernel: LustreError: 9717:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802975f7c00/0xf077f1a82d64763a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb91e15cd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 16:50:18 oak-gw06 kernel: LustreError: 9717:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 16:55:28 oak-gw06 kernel: LustreError: 9721:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d49f9cc0) refcount = 2 Aug 9 16:55:28 oak-gw06 kernel: LustreError: 9721:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 16:55:28 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 16:55:28 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 17:00:34 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 17:00:34 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 17:00:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502322934, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800854ad400/0xf077f1a82d64acd5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb91f09f4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 17:00:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 17:00:34 oak-gw06 kernel: LustreError: 9730:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801479b3d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 17:00:34 oak-gw06 kernel: LustreError: 9730:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 17:00:34 oak-gw06 kernel: LustreError: 9730:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801479b3d80) refcount = 2 Aug 9 17:00:34 oak-gw06 kernel: LustreError: 9730:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 17:00:34 oak-gw06 kernel: LustreError: 9730:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800854ad400/0xf077f1a82d64acd5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb91f09f4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 17:00:34 oak-gw06 kernel: LustreError: 9730:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 17:05:41 oak-gw06 kernel: LustreError: 9771:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803e8a20300) refcount = 2 Aug 9 17:05:41 oak-gw06 kernel: LustreError: 9771:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 17:05:41 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 17:05:41 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 17:10:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 17:10:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 17:10:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502323549, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800854ae600/0xf077f1a82d64dd26 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb91ffb5f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 17:10:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 17:15:59 oak-gw06 kernel: LustreError: 9789:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801479b3480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 17:15:59 oak-gw06 kernel: LustreError: 9789:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 17:15:59 oak-gw06 kernel: LustreError: 9789:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801479b3480) refcount = 2 Aug 9 17:15:59 oak-gw06 kernel: LustreError: 9789:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 17:16:00 oak-gw06 kernel: LustreError: 9789:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880142512000/0xf077f1a82d650670 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9207307 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 17:16:00 oak-gw06 kernel: LustreError: 9789:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 17:16:00 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 17:16:00 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 17:21:06 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 17:21:06 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 17:21:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502324166, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880136cf2e00/0xf077f1a82d6537d2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb920efbe expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 17:21:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 17:26:16 oak-gw06 kernel: LustreError: 9808:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802bb641c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 17:26:16 oak-gw06 kernel: LustreError: 9808:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bb641c00) refcount = 2 Aug 9 17:26:16 oak-gw06 kernel: LustreError: 9808:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 17:26:16 oak-gw06 kernel: LustreError: 9808:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88011ab0fc00/0xf077f1a82d657781 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9216720 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 17:26:16 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 17:26:16 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 17:31:21 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 17:31:21 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 17:31:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502324781, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88011ab0d000/0xf077f1a82d65a826 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb921e37c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 17:31:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 17:31:21 oak-gw06 kernel: LustreError: 9823:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880320fdd0c0) refcount = 2 Aug 9 17:31:21 oak-gw06 kernel: LustreError: 9823:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 17:36:28 oak-gw06 kernel: LustreError: 9832:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801c4358e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 17:36:28 oak-gw06 kernel: LustreError: 9832:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 17:36:28 oak-gw06 kernel: LustreError: 9832:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c4358e40) refcount = 2 Aug 9 17:36:28 oak-gw06 kernel: LustreError: 9832:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 17:36:28 oak-gw06 kernel: LustreError: 9832:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88041ef4cc00/0xf077f1a82d65e6d2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb92259c6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 17:36:28 oak-gw06 kernel: LustreError: 9832:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 17:36:28 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 17:36:28 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 17:41:38 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 17:41:38 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 17:41:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502325398, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803eb690000/0xf077f1a82d66389b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb922d64c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 17:41:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 17:41:38 oak-gw06 kernel: LustreError: 9843:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88026d9c0240) refcount = 2 Aug 9 17:41:38 oak-gw06 kernel: LustreError: 9843:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 17:46:48 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 17:46:48 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 17:51:56 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 17:51:56 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 17:51:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502326016, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801d16a7200/0xf077f1a82d66df21 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb923cb37 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 17:51:56 oak-gw06 kernel: LustreError: 9865:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802a36e7a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 17:51:56 oak-gw06 kernel: LustreError: 9865:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 17:51:56 oak-gw06 kernel: LustreError: 9865:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a36e7a80) refcount = 2 Aug 9 17:51:56 oak-gw06 kernel: LustreError: 9865:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 17:51:56 oak-gw06 kernel: LustreError: 9865:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801d16a7200/0xf077f1a82d66df21 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb923cb37 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 17:51:56 oak-gw06 kernel: LustreError: 9865:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 17:51:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 17:57:04 oak-gw06 kernel: LustreError: 9874:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88021070fe40) refcount = 2 Aug 9 17:57:04 oak-gw06 kernel: LustreError: 9874:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 17:57:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 17:57:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 18:02:10 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 18:02:10 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 18:02:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502326630, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801d3fc7a00/0xf077f1a82d6768a4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb924bfdc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 18:02:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 18:02:10 oak-gw06 kernel: LustreError: 9921:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88028e3fda80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 18:02:10 oak-gw06 kernel: LustreError: 9921:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 18:02:10 oak-gw06 kernel: LustreError: 9921:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88028e3fda80) refcount = 2 Aug 9 18:02:10 oak-gw06 kernel: LustreError: 9921:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 18:02:10 oak-gw06 kernel: LustreError: 9921:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801d3fc7a00/0xf077f1a82d6768a4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb924bfdc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 18:02:10 oak-gw06 kernel: LustreError: 9921:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 18:07:18 oak-gw06 kernel: LustreError: 9930:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801d0e429c0) refcount = 2 Aug 9 18:07:18 oak-gw06 kernel: LustreError: 9930:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 18:07:18 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 18:07:18 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 18:12:24 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 18:12:24 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 18:12:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502327244, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801952ec000/0xf077f1a82d683063 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb925b457 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 18:12:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 18:12:24 oak-gw06 kernel: LustreError: 9945:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802f3a26600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 18:12:24 oak-gw06 kernel: LustreError: 9945:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 18:12:24 oak-gw06 kernel: LustreError: 9945:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802f3a26600) refcount = 2 Aug 9 18:12:24 oak-gw06 kernel: LustreError: 9945:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 18:12:24 oak-gw06 kernel: LustreError: 9945:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801952ec000/0xf077f1a82d683063 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb925b457 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 18:12:24 oak-gw06 kernel: LustreError: 9945:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 18:17:34 oak-gw06 kernel: LustreError: 9949:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a7f72240) refcount = 2 Aug 9 18:17:34 oak-gw06 kernel: LustreError: 9949:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 18:17:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 18:17:34 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 18:22:39 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 18:22:39 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 18:22:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502327859, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88026485d400/0xf077f1a82d689699 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb926a903 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 18:22:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 18:27:49 oak-gw06 kernel: LustreError: 9965:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880301862f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 18:27:49 oak-gw06 kernel: LustreError: 9965:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 18:27:49 oak-gw06 kernel: LustreError: 9965:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880301862f00) refcount = 2 Aug 9 18:27:49 oak-gw06 kernel: LustreError: 9965:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 18:27:49 oak-gw06 kernel: LustreError: 9965:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880134311e00/0xf077f1a82d68bcef lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9272088 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 18:27:49 oak-gw06 kernel: LustreError: 9965:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 18:27:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 18:27:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 18:32:58 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 18:32:58 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 18:32:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502328478, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88020edd9200/0xf077f1a82d68e46b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9279e0a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 18:32:58 oak-gw06 kernel: LustreError: 9982:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803addcfcc0) refcount = 2 Aug 9 18:32:58 oak-gw06 kernel: LustreError: 9982:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 18:32:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 18:38:03 oak-gw06 kernel: LustreError: 9986:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88013481c000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 18:38:03 oak-gw06 kernel: LustreError: 9986:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 18:38:03 oak-gw06 kernel: LustreError: 9986:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88013481c000) refcount = 2 Aug 9 18:38:03 oak-gw06 kernel: LustreError: 9986:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 18:38:03 oak-gw06 kernel: LustreError: 9986:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801fd059200/0xf077f1a82d690b38 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb92815ea expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 18:38:03 oak-gw06 kernel: LustreError: 9986:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 18:38:03 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 18:38:03 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 18:43:10 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 18:43:10 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 18:43:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502329090, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801cdfd2a00/0xf077f1a82d693a24 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb928931f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 18:43:10 oak-gw06 kernel: LustreError: 10003:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880048623b40) refcount = 2 Aug 9 18:43:10 oak-gw06 kernel: LustreError: 10003:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 18:43:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 18:48:20 oak-gw06 kernel: LustreError: 10007:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880413e69600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 18:48:20 oak-gw06 kernel: LustreError: 10007:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 18:48:20 oak-gw06 kernel: LustreError: 10007:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e69600) refcount = 2 Aug 9 18:48:20 oak-gw06 kernel: LustreError: 10007:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 18:48:20 oak-gw06 kernel: LustreError: 10007:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880425bf7e00/0xf077f1a82d695f3f lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9290af8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 18:48:20 oak-gw06 kernel: LustreError: 10007:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 18:48:20 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 18:48:20 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 18:53:26 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 18:53:26 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 18:53:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502329706, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880345a15c00/0xf077f1a82d697cea lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb929886c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 18:53:26 oak-gw06 kernel: LustreError: 10023:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007c9facc0) refcount = 2 Aug 9 18:53:26 oak-gw06 kernel: LustreError: 10023:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 18:53:26 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 18:58:36 oak-gw06 kernel: LustreError: 10027:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803b37cc300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 18:58:36 oak-gw06 kernel: LustreError: 10027:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 18:58:36 oak-gw06 kernel: LustreError: 10027:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803b37cc300) refcount = 2 Aug 9 18:58:36 oak-gw06 kernel: LustreError: 10027:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 18:58:36 oak-gw06 kernel: LustreError: 10027:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880161552800/0xf077f1a82d69a3f6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb92a003e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 18:58:36 oak-gw06 kernel: LustreError: 10027:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 18:58:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 18:58:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 19:03:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 19:03:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 19:03:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502330322, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801214fc600/0xf077f1a82d69c80e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb92a7d3b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 19:03:42 oak-gw06 kernel: LustreError: 10075:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880426a00000) refcount = 2 Aug 9 19:03:42 oak-gw06 kernel: LustreError: 10075:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 19:03:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 19:08:51 oak-gw06 kernel: LustreError: 10079:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800195f93c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 19:08:51 oak-gw06 kernel: LustreError: 10079:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 19:08:51 oak-gw06 kernel: LustreError: 10079:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800195f93c0) refcount = 2 Aug 9 19:08:51 oak-gw06 kernel: LustreError: 10079:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 19:08:51 oak-gw06 kernel: LustreError: 10079:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88014902d000/0xf077f1a82d6a0038 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb92af545 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 19:08:51 oak-gw06 kernel: LustreError: 10079:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 19:08:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 19:08:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 19:13:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 19:13:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 19:13:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502330939, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88016f264600/0xf077f1a82d6a2f94 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb92b7250 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 19:13:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 19:13:59 oak-gw06 kernel: LustreError: 10095:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801a45b6300) refcount = 2 Aug 9 19:13:59 oak-gw06 kernel: LustreError: 10095:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 19:19:08 oak-gw06 kernel: LustreError: 10116:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802c8a9ba80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 19:19:08 oak-gw06 kernel: LustreError: 10116:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 19:19:08 oak-gw06 kernel: LustreError: 10116:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802c8a9ba80) refcount = 2 Aug 9 19:19:08 oak-gw06 kernel: LustreError: 10116:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 19:19:08 oak-gw06 kernel: LustreError: 10116:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801dd9a9e00/0xf077f1a82d6a70a8 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb92beb25 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 19:19:08 oak-gw06 kernel: LustreError: 10116:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 19:19:08 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 19:19:08 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 19:24:18 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 19:24:18 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 19:24:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502331558, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801245e3a00/0xf077f1a82d6bb4d1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb92c691e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 19:24:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 19:24:18 oak-gw06 kernel: LustreError: 10133:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88029be36900) refcount = 2 Aug 9 19:24:18 oak-gw06 kernel: LustreError: 10133:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 19:29:24 oak-gw06 kernel: LustreError: 10141:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880413e116c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 19:29:24 oak-gw06 kernel: LustreError: 10141:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 19:29:24 oak-gw06 kernel: LustreError: 10141:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413e116c0) refcount = 2 Aug 9 19:29:24 oak-gw06 kernel: LustreError: 10141:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 19:29:24 oak-gw06 kernel: LustreError: 10141:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88026d938c00/0xf077f1a82d6c1ba8 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb92ce14b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 19:29:24 oak-gw06 kernel: LustreError: 10141:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 19:29:24 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 19:29:24 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 19:34:32 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 19:34:32 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 19:34:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502332172, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88041e3da400/0xf077f1a82d6c9f51 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb92d5e25 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 19:34:32 oak-gw06 kernel: LustreError: 10164:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880195348240) refcount = 2 Aug 9 19:34:32 oak-gw06 kernel: LustreError: 10164:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 19:34:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 19:39:41 oak-gw06 kernel: LustreError: 10175:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802c7e76600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 19:39:41 oak-gw06 kernel: LustreError: 10175:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 19:39:41 oak-gw06 kernel: LustreError: 10175:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802c7e76600) refcount = 2 Aug 9 19:39:41 oak-gw06 kernel: LustreError: 10175:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 19:39:41 oak-gw06 kernel: LustreError: 10175:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880198dbd600/0xf077f1a82d6d8c55 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb92dd61a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 19:39:41 oak-gw06 kernel: LustreError: 10175:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 19:39:41 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 19:39:41 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 19:44:47 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 19:44:47 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 19:44:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502332787, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801c052f600/0xf077f1a82d6dde2c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb92e52d1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 19:44:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 19:44:47 oak-gw06 kernel: LustreError: 10186:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800ac1c8240) refcount = 2 Aug 9 19:44:47 oak-gw06 kernel: LustreError: 10186:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 19:49:55 oak-gw06 kernel: LustreError: 10194:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88038cf66180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 19:49:55 oak-gw06 kernel: LustreError: 10194:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 19:49:55 oak-gw06 kernel: LustreError: 10194:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88038cf66180) refcount = 2 Aug 9 19:49:55 oak-gw06 kernel: LustreError: 10194:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 19:49:55 oak-gw06 kernel: LustreError: 10194:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88026394a400/0xf077f1a82d6e15d1 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb92eca87 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 19:49:55 oak-gw06 kernel: LustreError: 10194:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 19:49:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 19:49:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 19:55:04 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 19:55:04 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 19:55:04 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502333404, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801b3b5a400/0xf077f1a82d6e4e4f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb92f4864 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 19:55:04 oak-gw06 kernel: LustreError: 10210:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d3ae6600) refcount = 2 Aug 9 19:55:04 oak-gw06 kernel: LustreError: 10210:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 19:55:04 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 20:00:10 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 20:00:10 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 20:05:18 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 20:05:18 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 20:05:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502334018, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802d2aad800/0xf077f1a82d6edca9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9303c5a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 20:05:18 oak-gw06 kernel: LustreError: 10259:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800145c6900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 20:05:18 oak-gw06 kernel: LustreError: 10259:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 20:05:18 oak-gw06 kernel: LustreError: 10259:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800145c6900) refcount = 2 Aug 9 20:05:18 oak-gw06 kernel: LustreError: 10259:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 20:05:18 oak-gw06 kernel: LustreError: 10259:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802d2aad800/0xf077f1a82d6edca9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9303c5a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 20:05:18 oak-gw06 kernel: LustreError: 10259:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 20:05:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 20:10:25 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 20:10:25 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 20:15:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 20:15:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 20:15:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502334631, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800b2486000/0xf077f1a82d6f5b9e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb931324f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 20:15:31 oak-gw06 kernel: LustreError: 10282:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801f7e0dc00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 20:15:31 oak-gw06 kernel: LustreError: 10282:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801f7e0dc00) refcount = 2 Aug 9 20:15:31 oak-gw06 kernel: LustreError: 10282:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 20:15:31 oak-gw06 kernel: LustreError: 10282:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800b2486000/0xf077f1a82d6f5b9e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb931324f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 20:15:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 20:20:41 oak-gw06 kernel: LustreError: 10298:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bb27c0c0) refcount = 2 Aug 9 20:20:41 oak-gw06 kernel: LustreError: 10298:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 20:20:41 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 20:20:41 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 20:25:47 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 20:25:47 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 20:25:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502335247, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801958ff600/0xf077f1a82d6fdebb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9322756 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 20:25:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 20:25:47 oak-gw06 kernel: LustreError: 10303:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88014c884180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 20:25:47 oak-gw06 kernel: LustreError: 10303:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 20:25:47 oak-gw06 kernel: LustreError: 10303:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88014c884180) refcount = 2 Aug 9 20:25:47 oak-gw06 kernel: LustreError: 10303:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 20:25:47 oak-gw06 kernel: LustreError: 10303:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801958ff600/0xf077f1a82d6fdebb lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9322756 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 20:25:47 oak-gw06 kernel: LustreError: 10303:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 20:30:56 oak-gw06 kernel: LustreError: 10315:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801bdb44d80) refcount = 1 Aug 9 20:30:56 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 20:30:56 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 20:36:05 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 20:36:05 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 20:36:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502335865, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800392b0600/0xf077f1a82d6ffd23 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9331c48 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 20:36:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 20:36:05 oak-gw06 kernel: LustreError: 10317:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801bdb44a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 20:36:05 oak-gw06 kernel: LustreError: 10317:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 20:36:05 oak-gw06 kernel: LustreError: 10317:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801bdb44a80) refcount = 2 Aug 9 20:36:05 oak-gw06 kernel: LustreError: 10317:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 20:36:05 oak-gw06 kernel: LustreError: 10317:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800392b0600/0xf077f1a82d6ffd23 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9331c48 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 20:41:10 oak-gw06 kernel: LustreError: 10332:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802abe7dd80) refcount = 2 Aug 9 20:41:10 oak-gw06 kernel: LustreError: 10332:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 20:41:10 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 20:41:10 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 20:46:19 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 20:46:19 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 20:46:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502336479, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800392b0200/0xf077f1a82d7028f1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb93410f4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 20:46:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 20:46:19 oak-gw06 kernel: LustreError: 10341:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802abe7dd80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 20:46:19 oak-gw06 kernel: LustreError: 10341:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 20:46:19 oak-gw06 kernel: LustreError: 10341:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802abe7dd80) refcount = 2 Aug 9 20:46:19 oak-gw06 kernel: LustreError: 10341:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 20:46:19 oak-gw06 kernel: LustreError: 10341:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800392b0200/0xf077f1a82d7028f1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb93410f4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 20:46:19 oak-gw06 kernel: LustreError: 10341:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 20:51:29 oak-gw06 kernel: LustreError: 10365:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880407f1dc00) refcount = 2 Aug 9 20:51:29 oak-gw06 kernel: LustreError: 10365:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 20:51:29 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 20:51:29 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 20:56:36 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 20:56:36 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 20:56:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502337096, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880130ba4400/0xf077f1a82d7169a8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9350767 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 20:56:36 oak-gw06 kernel: LustreError: 10369:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88041cace240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 20:56:36 oak-gw06 kernel: LustreError: 10369:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 20:56:36 oak-gw06 kernel: LustreError: 10369:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041cace240) refcount = 2 Aug 9 20:56:36 oak-gw06 kernel: LustreError: 10369:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 20:56:36 oak-gw06 kernel: LustreError: 10369:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880130ba4400/0xf077f1a82d7169a8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9350767 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 20:56:36 oak-gw06 kernel: LustreError: 10369:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 20:56:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 21:01:46 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 21:01:46 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 21:06:52 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 21:06:52 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 21:06:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502337712, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880161552600/0xf077f1a82d7196b8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb935fdda expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 21:06:52 oak-gw06 kernel: LustreError: 10418:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88022e412e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 21:06:52 oak-gw06 kernel: LustreError: 10418:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022e412e40) refcount = 2 Aug 9 21:06:52 oak-gw06 kernel: LustreError: 10418:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 21:06:52 oak-gw06 kernel: LustreError: 10418:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880161552600/0xf077f1a82d7196b8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb935fdda expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 21:06:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 21:11:59 oak-gw06 kernel: LustreError: 10432:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801d6915900) refcount = 2 Aug 9 21:11:59 oak-gw06 kernel: LustreError: 10432:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 21:11:59 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 21:11:59 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 21:17:07 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 21:17:07 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 21:17:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502338327, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88011f270a00/0xf077f1a82d71c3cf lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb936f35f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 21:17:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 21:22:16 oak-gw06 kernel: LustreError: 10445:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802a0ee1300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 21:22:16 oak-gw06 kernel: LustreError: 10445:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 21:22:16 oak-gw06 kernel: LustreError: 10445:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a0ee1300) refcount = 2 Aug 9 21:22:16 oak-gw06 kernel: LustreError: 10445:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 21:22:16 oak-gw06 kernel: LustreError: 10445:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8804042a9200/0xf077f1a82d71e643 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9376b46 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 21:22:16 oak-gw06 kernel: LustreError: 10445:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 21:22:16 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 21:22:16 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 21:27:23 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 21:27:23 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 21:27:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502338943, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8804042a8600/0xf077f1a82d72097b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb937e858 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 21:27:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 21:32:28 oak-gw06 kernel: LustreError: 10466:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880406fc90c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 21:32:28 oak-gw06 kernel: LustreError: 10466:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880406fc90c0) refcount = 2 Aug 9 21:32:28 oak-gw06 kernel: LustreError: 10466:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 21:32:28 oak-gw06 kernel: LustreError: 10466:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88013da12c00/0xf077f1a82d7223ad lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9385f19 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 21:32:28 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 21:32:28 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 21:37:34 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 21:37:34 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 21:37:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502339554, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801ce588600/0xf077f1a82d7247da lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb938dc55 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 21:37:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 21:37:34 oak-gw06 kernel: LustreError: 10474:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801f1146000) refcount = 2 Aug 9 21:37:34 oak-gw06 kernel: LustreError: 10474:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 21:42:43 oak-gw06 kernel: LustreError: 10486:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880043847f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 21:42:43 oak-gw06 kernel: LustreError: 10486:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 21:42:43 oak-gw06 kernel: LustreError: 10486:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880043847f00) refcount = 2 Aug 9 21:42:43 oak-gw06 kernel: LustreError: 10486:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 21:42:43 oak-gw06 kernel: LustreError: 10486:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803f5ba7a00/0xf077f1a82d727028 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9395371 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 21:42:43 oak-gw06 kernel: LustreError: 10486:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 21:42:43 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 21:42:43 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 21:47:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 21:47:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 21:47:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502340169, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803f5ba5400/0xf077f1a82d729600 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb939d075 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 21:47:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 21:47:49 oak-gw06 kernel: LustreError: 10490:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802e3644900) refcount = 2 Aug 9 21:47:49 oak-gw06 kernel: LustreError: 10490:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 21:52:55 oak-gw06 kernel: LustreError: 10505:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880406bc8a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 21:52:55 oak-gw06 kernel: LustreError: 10505:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 21:52:55 oak-gw06 kernel: LustreError: 10505:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880406bc8a80) refcount = 2 Aug 9 21:52:55 oak-gw06 kernel: LustreError: 10505:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 21:52:55 oak-gw06 kernel: LustreError: 10505:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800a8e8c400/0xf077f1a82d72bd6e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb93a47c9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 21:52:55 oak-gw06 kernel: LustreError: 10505:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 21:52:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 21:52:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 21:58:05 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 21:58:05 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 21:58:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502340785, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801233b7200/0xf077f1a82d72e958 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb93ac678 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 21:58:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 21:58:05 oak-gw06 kernel: LustreError: 10514:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803b5fc46c0) refcount = 2 Aug 9 21:58:05 oak-gw06 kernel: LustreError: 10514:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 22:03:16 oak-gw06 kernel: LustreError: 10559:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880399d64840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 22:03:16 oak-gw06 kernel: LustreError: 10559:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 22:03:16 oak-gw06 kernel: LustreError: 10559:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880399d64840) refcount = 2 Aug 9 22:03:16 oak-gw06 kernel: LustreError: 10559:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 22:03:16 oak-gw06 kernel: LustreError: 10559:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880142512800/0xf077f1a82d731d61 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb93b3e19 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 22:03:16 oak-gw06 kernel: LustreError: 10559:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 22:03:16 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 22:03:16 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 22:08:24 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 22:08:24 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 22:08:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502341404, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880327487800/0xf077f1a82d734aa9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb93bbc51 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 22:08:24 oak-gw06 kernel: LustreError: 10563:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801e08bdb40) refcount = 2 Aug 9 22:08:24 oak-gw06 kernel: LustreError: 10563:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 22:08:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 22:13:31 oak-gw06 kernel: LustreError: 10578:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802867476c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 22:13:31 oak-gw06 kernel: LustreError: 10578:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 22:13:31 oak-gw06 kernel: LustreError: 10578:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802867476c0) refcount = 2 Aug 9 22:13:31 oak-gw06 kernel: LustreError: 10578:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 22:13:31 oak-gw06 kernel: LustreError: 10578:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802674f2800/0xf077f1a82d73674a lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb93c33c1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 22:13:31 oak-gw06 kernel: LustreError: 10578:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 22:13:31 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 22:13:31 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 22:18:40 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 22:18:40 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 22:18:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502342020, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802674f2800/0xf077f1a82d73762a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb93cb190 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 22:18:40 oak-gw06 kernel: LustreError: 10582:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801d6915240) refcount = 2 Aug 9 22:18:40 oak-gw06 kernel: LustreError: 10582:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 22:18:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 22:23:46 oak-gw06 kernel: LustreError: 10594:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801e08bdd80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 22:23:46 oak-gw06 kernel: LustreError: 10594:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 22:23:46 oak-gw06 kernel: LustreError: 10594:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801e08bdd80) refcount = 2 Aug 9 22:23:46 oak-gw06 kernel: LustreError: 10594:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 22:23:46 oak-gw06 kernel: LustreError: 10594:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88025d26e400/0xf077f1a82d738d84 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb93d2804 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 22:23:46 oak-gw06 kernel: LustreError: 10594:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 22:23:46 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 22:23:46 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 22:28:56 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 22:28:56 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 22:28:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502342636, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802d2aae000/0xf077f1a82d73a8b2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb93da5d3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 22:28:56 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 22:28:56 oak-gw06 kernel: LustreError: 10598:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88009fefb9c0) refcount = 2 Aug 9 22:28:56 oak-gw06 kernel: LustreError: 10598:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 22:34:02 oak-gw06 kernel: LustreError: 10613:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88011063a9c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 22:34:02 oak-gw06 kernel: LustreError: 10613:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 22:34:02 oak-gw06 kernel: LustreError: 10613:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88011063a9c0) refcount = 2 Aug 9 22:34:02 oak-gw06 kernel: LustreError: 10613:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 22:34:02 oak-gw06 kernel: LustreError: 10613:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88014982bc00/0xf077f1a82d73c7c2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb93e1b98 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 22:34:02 oak-gw06 kernel: LustreError: 10613:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 22:34:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 22:34:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 22:39:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 22:39:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 22:39:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502343253, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88028f7bf400/0xf077f1a82d73e24f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb93e9ac5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 22:39:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 22:39:13 oak-gw06 kernel: LustreError: 10625:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801d6915840) refcount = 2 Aug 9 22:39:13 oak-gw06 kernel: LustreError: 10625:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 22:44:25 oak-gw06 kernel: LustreError: 10645:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802bb211180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 22:44:25 oak-gw06 kernel: LustreError: 10645:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 22:44:25 oak-gw06 kernel: LustreError: 10645:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bb211180) refcount = 2 Aug 9 22:44:25 oak-gw06 kernel: LustreError: 10645:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 22:44:25 oak-gw06 kernel: LustreError: 10645:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801d0ed7600/0xf077f1a82d74ba0d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb93f139a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 22:44:25 oak-gw06 kernel: LustreError: 10645:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 22:44:25 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 22:44:25 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 22:49:38 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 22:49:38 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 22:49:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502343878, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801438d5800/0xf077f1a82d75332f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb93f925e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 22:49:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 22:49:38 oak-gw06 kernel: LustreError: 10649:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880397c04780) refcount = 2 Aug 9 22:49:38 oak-gw06 kernel: LustreError: 10649:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 22:54:50 oak-gw06 kernel: LustreError: 10660:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88041923d780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 22:54:50 oak-gw06 kernel: LustreError: 10660:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 22:54:50 oak-gw06 kernel: LustreError: 10660:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041923d780) refcount = 2 Aug 9 22:54:50 oak-gw06 kernel: LustreError: 10660:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 22:54:50 oak-gw06 kernel: LustreError: 10660:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88022fd97200/0xf077f1a82d754e1e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9400b79 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 22:54:50 oak-gw06 kernel: LustreError: 10660:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 22:54:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 22:54:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 22:59:58 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 22:59:58 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 22:59:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502344498, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88010ccf4400/0xf077f1a82d756976 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb94088ed expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 22:59:58 oak-gw06 kernel: LustreError: 10668:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041bc723c0) refcount = 2 Aug 9 22:59:58 oak-gw06 kernel: LustreError: 10668:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 22:59:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 23:05:05 oak-gw06 kernel: LustreError: 10712:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803307309c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 23:05:05 oak-gw06 kernel: LustreError: 10712:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 23:05:05 oak-gw06 kernel: LustreError: 10712:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803307309c0) refcount = 2 Aug 9 23:05:05 oak-gw06 kernel: LustreError: 10712:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 23:05:05 oak-gw06 kernel: LustreError: 10712:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802d0236e00/0xf077f1a82d75832a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb94101ad expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 23:05:05 oak-gw06 kernel: LustreError: 10712:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 23:05:05 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 23:05:05 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 23:10:14 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 23:10:14 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 23:10:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502345114, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88015f644600/0xf077f1a82d759de8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9417f6e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 23:10:14 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 23:10:14 oak-gw06 kernel: LustreError: 10724:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801db758f00) refcount = 2 Aug 9 23:10:14 oak-gw06 kernel: LustreError: 10724:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 23:15:23 oak-gw06 kernel: LustreError: 10732:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880090f8e900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 23:15:23 oak-gw06 kernel: LustreError: 10732:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 23:15:23 oak-gw06 kernel: LustreError: 10732:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880090f8e900) refcount = 2 Aug 9 23:15:23 oak-gw06 kernel: LustreError: 10732:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 23:15:23 oak-gw06 kernel: LustreError: 10732:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88015f647a00/0xf077f1a82d75bcb2 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb941f7f6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 23:15:23 oak-gw06 kernel: LustreError: 10732:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 23:15:23 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 23:15:23 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 23:20:30 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 23:20:30 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 23:20:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502345730, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88012491d400/0xf077f1a82d75e458 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb942744b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 23:20:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 23:20:30 oak-gw06 kernel: LustreError: 10743:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413f39780) refcount = 2 Aug 9 23:20:30 oak-gw06 kernel: LustreError: 10743:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 23:25:41 oak-gw06 kernel: LustreError: 10754:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88042c239d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 23:25:41 oak-gw06 kernel: LustreError: 10754:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 23:25:41 oak-gw06 kernel: LustreError: 10754:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042c239d80) refcount = 2 Aug 9 23:25:41 oak-gw06 kernel: LustreError: 10754:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 23:25:41 oak-gw06 kernel: LustreError: 10754:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88012491e600/0xf077f1a82d760c6e lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb942ee77 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 23:25:41 oak-gw06 kernel: LustreError: 10754:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 23:25:41 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 23:25:41 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 23:30:48 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 23:30:48 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 23:30:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502346348, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88013da10a00/0xf077f1a82d7630e8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9436b5f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 23:30:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 23:30:48 oak-gw06 kernel: LustreError: 10767:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880274ba7780) refcount = 2 Aug 9 23:30:48 oak-gw06 kernel: LustreError: 10767:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 23:35:54 oak-gw06 kernel: LustreError: 10775:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88009fefb840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 23:35:54 oak-gw06 kernel: LustreError: 10775:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 23:35:54 oak-gw06 kernel: LustreError: 10775:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88009fefb840) refcount = 2 Aug 9 23:35:54 oak-gw06 kernel: LustreError: 10775:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 23:35:54 oak-gw06 kernel: LustreError: 10775:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880011c64200/0xf077f1a82d765c9a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb943e2ba expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 23:35:54 oak-gw06 kernel: LustreError: 10775:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 23:35:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 23:35:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 23:41:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 23:41:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 23:41:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502346960, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801da3b5000/0xf077f1a82d7686d9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9445f9b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 23:41:00 oak-gw06 kernel: LustreError: 10786:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800b8d34000) refcount = 2 Aug 9 23:41:00 oak-gw06 kernel: LustreError: 10786:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 23:41:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 23:46:11 oak-gw06 kernel: LustreError: 10791:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880127cbdcc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 23:46:11 oak-gw06 kernel: LustreError: 10791:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 23:46:11 oak-gw06 kernel: LustreError: 10791:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880127cbdcc0) refcount = 2 Aug 9 23:46:11 oak-gw06 kernel: LustreError: 10791:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 23:46:11 oak-gw06 kernel: LustreError: 10791:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88027f715000/0xf077f1a82d76ad91 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb944d838 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 23:46:11 oak-gw06 kernel: LustreError: 10791:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 23:46:11 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 23:46:11 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 9 23:51:21 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 9 23:51:21 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 9 23:51:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502347581, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880172f34e00/0xf077f1a82d76ce4c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb945569a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 23:51:21 oak-gw06 kernel: LustreError: 10806:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880222ccd900) refcount = 2 Aug 9 23:51:21 oak-gw06 kernel: LustreError: 10806:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 23:51:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 9 23:56:30 oak-gw06 kernel: LustreError: 10823:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801b7f05a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 9 23:56:30 oak-gw06 kernel: LustreError: 10823:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 9 23:56:30 oak-gw06 kernel: LustreError: 10823:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801b7f05a80) refcount = 2 Aug 9 23:56:30 oak-gw06 kernel: LustreError: 10823:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 9 23:56:30 oak-gw06 kernel: LustreError: 10823:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88020a59d200/0xf077f1a82d76eeeb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb945d05d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 9 23:56:30 oak-gw06 kernel: LustreError: 10823:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 9 23:56:30 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 9 23:56:30 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 00:01:36 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 00:01:36 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 00:01:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502348196, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880072f27c00/0xf077f1a82d76fedc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9464c11 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 00:01:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 00:01:36 oak-gw06 kernel: LustreError: 10868:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801279afb40) refcount = 2 Aug 10 00:01:36 oak-gw06 kernel: LustreError: 10868:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 00:06:44 oak-gw06 kernel: LustreError: 10872:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803443d7c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 00:06:44 oak-gw06 kernel: LustreError: 10872:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 00:06:44 oak-gw06 kernel: LustreError: 10872:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803443d7c00) refcount = 2 Aug 10 00:06:44 oak-gw06 kernel: LustreError: 10872:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 00:06:44 oak-gw06 kernel: LustreError: 10872:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800436f8400/0xf077f1a82d7704c4 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb946c3c0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 00:06:44 oak-gw06 kernel: LustreError: 10872:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 00:06:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 00:06:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 00:11:54 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 00:11:54 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 00:11:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502348814, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88020eddae00/0xf077f1a82d770901 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb94741c7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 00:11:54 oak-gw06 kernel: LustreError: 10885:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803443d7e40) refcount = 2 Aug 10 00:11:54 oak-gw06 kernel: LustreError: 10885:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 00:11:54 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 00:17:00 oak-gw06 kernel: LustreError: 10889:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803443d7c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 00:17:00 oak-gw06 kernel: LustreError: 10889:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 00:17:00 oak-gw06 kernel: LustreError: 10889:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803443d7c00) refcount = 2 Aug 10 00:17:00 oak-gw06 kernel: LustreError: 10889:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 00:17:00 oak-gw06 kernel: LustreError: 10889:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88020edd9800/0xf077f1a82d770c5e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb947b93e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 00:17:00 oak-gw06 kernel: LustreError: 10889:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 00:17:00 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 00:17:00 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 00:22:09 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 00:22:09 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 00:22:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502349429, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88020edd8000/0xf077f1a82d7713a4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9483714 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 00:22:09 oak-gw06 kernel: LustreError: 10900:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803443d7a80) refcount = 2 Aug 10 00:22:09 oak-gw06 kernel: LustreError: 10900:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 00:22:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 00:27:16 oak-gw06 kernel: LustreError: 10908:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801c1ff6a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 00:27:16 oak-gw06 kernel: LustreError: 10908:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 00:27:16 oak-gw06 kernel: LustreError: 10908:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c1ff6a80) refcount = 2 Aug 10 00:27:16 oak-gw06 kernel: LustreError: 10908:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 00:27:16 oak-gw06 kernel: LustreError: 10908:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88020edd8000/0xf077f1a82d7727d9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb948ad26 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 00:27:16 oak-gw06 kernel: LustreError: 10908:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 00:27:16 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 00:27:16 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 00:32:23 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 00:32:23 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 00:32:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502350043, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88020a59e200/0xf077f1a82d7753df lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9492abd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 00:32:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 00:32:23 oak-gw06 kernel: LustreError: 10924:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88024a190840) refcount = 2 Aug 10 00:32:23 oak-gw06 kernel: LustreError: 10924:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 00:37:31 oak-gw06 kernel: LustreError: 10928:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8804186cb300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 00:37:31 oak-gw06 kernel: LustreError: 10928:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 00:37:31 oak-gw06 kernel: LustreError: 10928:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8804186cb300) refcount = 2 Aug 10 00:37:31 oak-gw06 kernel: LustreError: 10928:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 00:37:31 oak-gw06 kernel: LustreError: 10928:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88012e55a400/0xf077f1a82d779085 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb949a074 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 00:37:31 oak-gw06 kernel: LustreError: 10928:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 00:37:31 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 00:37:31 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 00:42:39 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 00:42:39 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 00:42:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502350659, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802de581400/0xf077f1a82d77d391 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb94a1dd3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 00:42:39 oak-gw06 kernel: LustreError: 10942:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800b17e3180) refcount = 2 Aug 10 00:42:39 oak-gw06 kernel: LustreError: 10942:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 00:42:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 00:47:50 oak-gw06 kernel: LustreError: 10960:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880407f1d0c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 00:47:50 oak-gw06 kernel: LustreError: 10960:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 00:47:50 oak-gw06 kernel: LustreError: 10960:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880407f1d0c0) refcount = 2 Aug 10 00:47:50 oak-gw06 kernel: LustreError: 10960:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 00:47:50 oak-gw06 kernel: LustreError: 10960:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801e2d92c00/0xf077f1a82d7874ad lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb94a9693 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 00:47:50 oak-gw06 kernel: LustreError: 10960:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 00:47:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 00:47:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 00:53:01 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 00:53:01 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 00:53:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502351281, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801f02da600/0xf077f1a82d7908da lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb94b1397 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 00:53:01 oak-gw06 kernel: LustreError: 10975:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802823c8840) refcount = 2 Aug 10 00:53:01 oak-gw06 kernel: LustreError: 10975:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 00:53:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 00:58:07 oak-gw06 kernel: LustreError: 10987:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88021b3796c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 00:58:07 oak-gw06 kernel: LustreError: 10987:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 00:58:07 oak-gw06 kernel: LustreError: 10987:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88021b3796c0) refcount = 2 Aug 10 00:58:07 oak-gw06 kernel: LustreError: 10987:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 00:58:07 oak-gw06 kernel: LustreError: 10987:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803bc69e600/0xf077f1a82d797599 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb94b8b4d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 00:58:07 oak-gw06 kernel: LustreError: 10987:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 00:58:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 00:58:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 01:03:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 01:03:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 01:03:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502351893, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88014982ae00/0xf077f1a82d7a150a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb94c06ec expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 01:03:13 oak-gw06 kernel: LustreError: 11038:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801279afe40) refcount = 2 Aug 10 01:03:13 oak-gw06 kernel: LustreError: 11038:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 01:03:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 01:08:22 oak-gw06 kernel: LustreError: 11046:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800ac338300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 01:08:22 oak-gw06 kernel: LustreError: 11046:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 01:08:22 oak-gw06 kernel: LustreError: 11046:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800ac338300) refcount = 2 Aug 10 01:08:22 oak-gw06 kernel: LustreError: 11046:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 01:08:22 oak-gw06 kernel: LustreError: 11046:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88026246c400/0xf077f1a82d7a5a07 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb94c7e5c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 01:08:22 oak-gw06 kernel: LustreError: 11046:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 01:08:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 01:08:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 01:13:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 01:13:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 01:13:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502352511, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802d9fe0a00/0xf077f1a82d7a85f8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb94cfb59 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 01:13:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 01:13:31 oak-gw06 kernel: LustreError: 11062:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88024e23e180) refcount = 2 Aug 10 01:13:31 oak-gw06 kernel: LustreError: 11062:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 01:18:47 oak-gw06 kernel: LustreError: 11066:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801378bb240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 01:18:47 oak-gw06 kernel: LustreError: 11066:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 01:18:47 oak-gw06 kernel: LustreError: 11066:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801378bb240) refcount = 2 Aug 10 01:18:47 oak-gw06 kernel: LustreError: 11066:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 01:18:47 oak-gw06 kernel: LustreError: 11066:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880282d46e00/0xf077f1a82d7ae95d lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb94d742e expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 01:18:47 oak-gw06 kernel: LustreError: 11066:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 01:18:47 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 01:18:47 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 01:23:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 01:23:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 01:23:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502353139, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803db350e00/0xf077f1a82d7b1341 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb94df1b0 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 01:23:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 01:23:59 oak-gw06 kernel: LustreError: 11082:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88024e23e540) refcount = 2 Aug 10 01:23:59 oak-gw06 kernel: LustreError: 11082:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 01:29:05 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 01:29:05 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 01:34:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 01:34:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 01:34:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502353751, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8804277d8200/0xf077f1a82d7c7f8e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb94ee368 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 01:34:11 oak-gw06 kernel: LustreError: 11166:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88010c2e36c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 01:34:11 oak-gw06 kernel: LustreError: 11166:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 01:34:11 oak-gw06 kernel: LustreError: 11166:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88010c2e36c0) refcount = 2 Aug 10 01:34:11 oak-gw06 kernel: LustreError: 11166:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 01:34:11 oak-gw06 kernel: LustreError: 11166:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8804277d8200/0xf077f1a82d7c7f8e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb94ee368 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 01:34:11 oak-gw06 kernel: LustreError: 11166:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 01:34:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 01:39:22 oak-gw06 kernel: LustreError: 11171:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff88042b6003c0) refcount = 2 Aug 10 01:39:22 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 01:39:22 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 01:44:34 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 01:44:34 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 01:44:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502354374, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803a87de800/0xf077f1a82d80f4e9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb94fd9bf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 01:44:34 oak-gw06 kernel: LustreError: 11181:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88042b600180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 01:44:34 oak-gw06 kernel: LustreError: 11181:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 01:44:34 oak-gw06 kernel: LustreError: 11181:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042b600180) refcount = 2 Aug 10 01:44:34 oak-gw06 kernel: LustreError: 11181:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 01:44:34 oak-gw06 kernel: LustreError: 11181:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803a87de800/0xf077f1a82d80f4e9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb94fd9bf expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 01:44:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 01:49:43 oak-gw06 kernel: LustreError: 11184:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042b490180) refcount = 1 Aug 10 01:49:43 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 01:49:43 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 01:54:52 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 01:54:52 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 01:54:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502354992, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880118c5c800/0xf077f1a82d811a43 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb950cee2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 01:54:52 oak-gw06 kernel: LustreError: 11200:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88016a68dcc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 01:54:52 oak-gw06 kernel: LustreError: 11200:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 01:54:52 oak-gw06 kernel: LustreError: 11200:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88016a68dcc0) refcount = 2 Aug 10 01:54:52 oak-gw06 kernel: LustreError: 11200:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 01:54:52 oak-gw06 kernel: LustreError: 11200:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880118c5c800/0xf077f1a82d811a43 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb950cee2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 01:54:52 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 02:00:02 oak-gw06 kernel: LustreError: 11211:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88011063a780) refcount = 2 Aug 10 02:00:02 oak-gw06 kernel: LustreError: 11211:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 02:00:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 02:00:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 02:05:09 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 02:05:09 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 02:05:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502355609, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88022a841600/0xf077f1a82d81664e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb951c229 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 02:05:09 oak-gw06 kernel: LustreError: 11261:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802d9d58c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 02:05:09 oak-gw06 kernel: LustreError: 11261:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 02:05:09 oak-gw06 kernel: LustreError: 11261:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802d9d58c00) refcount = 2 Aug 10 02:05:09 oak-gw06 kernel: LustreError: 11261:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 02:05:09 oak-gw06 kernel: LustreError: 11261:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88022a841600/0xf077f1a82d81664e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb951c229 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 02:05:09 oak-gw06 kernel: LustreError: 11261:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 02:05:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 02:10:19 oak-gw06 kernel: LustreError: 11276:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8804266f4cc0) refcount = 2 Aug 10 02:10:19 oak-gw06 kernel: LustreError: 11276:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 02:10:19 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 02:10:19 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 02:15:27 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 02:15:27 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 02:15:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502356227, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803c365fe00/0xf077f1a82d827f58 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb952b5bd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 02:15:27 oak-gw06 kernel: LustreError: 11280:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88018df52a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 02:15:27 oak-gw06 kernel: LustreError: 11280:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 02:15:27 oak-gw06 kernel: LustreError: 11280:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88018df52a80) refcount = 2 Aug 10 02:15:27 oak-gw06 kernel: LustreError: 11280:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 02:15:27 oak-gw06 kernel: LustreError: 11280:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803c365fe00/0xf077f1a82d827f58 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb952b5bd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 02:15:27 oak-gw06 kernel: LustreError: 11280:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 02:15:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 02:20:37 oak-gw06 kernel: LustreError: 11290:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801d363f0c0) refcount = 2 Aug 10 02:20:37 oak-gw06 kernel: LustreError: 11290:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 02:20:37 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 02:20:37 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 02:25:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 02:25:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 02:25:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502356844, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88013d562e00/0xf077f1a82d82b329 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb953a919 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 02:25:44 oak-gw06 kernel: LustreError: 11294:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801aa736e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 02:25:44 oak-gw06 kernel: LustreError: 11294:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 02:25:44 oak-gw06 kernel: LustreError: 11294:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801aa736e40) refcount = 2 Aug 10 02:25:44 oak-gw06 kernel: LustreError: 11294:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 02:25:44 oak-gw06 kernel: LustreError: 11294:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88013d562e00/0xf077f1a82d82b329 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb953a919 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 02:25:44 oak-gw06 kernel: LustreError: 11294:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 02:25:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 02:30:55 oak-gw06 kernel: LustreError: 11310:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880283643900) refcount = 2 Aug 10 02:30:55 oak-gw06 kernel: LustreError: 11310:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 02:30:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 02:30:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 02:36:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 02:36:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 02:36:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502357462, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801c23f6e00/0xf077f1a82d82e9e0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9549eba expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 02:36:02 oak-gw06 kernel: LustreError: 11314:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801fd31a3c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 02:36:02 oak-gw06 kernel: LustreError: 11314:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 02:36:02 oak-gw06 kernel: LustreError: 11314:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801fd31a3c0) refcount = 2 Aug 10 02:36:02 oak-gw06 kernel: LustreError: 11314:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 02:36:02 oak-gw06 kernel: LustreError: 11314:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801c23f6e00/0xf077f1a82d82e9e0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9549eba expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 02:36:02 oak-gw06 kernel: LustreError: 11314:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 02:36:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 02:41:11 oak-gw06 kernel: LustreError: 11326:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800b5b74480) refcount = 2 Aug 10 02:41:11 oak-gw06 kernel: LustreError: 11326:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 02:41:11 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 02:41:11 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 02:46:19 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 02:46:19 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 02:46:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502358079, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801fcc82c00/0xf077f1a82d831775 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9559320 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 02:46:19 oak-gw06 kernel: LustreError: 11334:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880135fdbb40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 02:46:19 oak-gw06 kernel: LustreError: 11334:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 02:46:19 oak-gw06 kernel: LustreError: 11334:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880135fdbb40) refcount = 2 Aug 10 02:46:19 oak-gw06 kernel: LustreError: 11334:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 02:46:19 oak-gw06 kernel: LustreError: 11334:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801fcc82c00/0xf077f1a82d831775 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9559320 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 02:46:19 oak-gw06 kernel: LustreError: 11334:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 02:46:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 02:51:30 oak-gw06 kernel: LustreError: 11346:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802f3a26780) refcount = 2 Aug 10 02:51:30 oak-gw06 kernel: LustreError: 11346:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 02:51:30 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 02:51:30 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 02:56:39 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 02:56:39 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 02:56:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502358699, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880090ca2400/0xf077f1a82d8349be lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9568977 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 02:56:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 02:56:39 oak-gw06 kernel: LustreError: 11350:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880421a93240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 02:56:39 oak-gw06 kernel: LustreError: 11350:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 02:56:39 oak-gw06 kernel: LustreError: 11350:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880421a93240) refcount = 2 Aug 10 02:56:39 oak-gw06 kernel: LustreError: 11350:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 02:56:39 oak-gw06 kernel: LustreError: 11350:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880090ca2400/0xf077f1a82d8349be lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9568977 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 02:56:39 oak-gw06 kernel: LustreError: 11350:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 03:01:44 oak-gw06 kernel: LustreError: 11397:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bb641480) refcount = 2 Aug 10 03:01:44 oak-gw06 kernel: LustreError: 11397:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 03:01:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 03:01:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 03:06:53 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 03:06:53 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 03:06:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502359313, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803b5bce000/0xf077f1a82d8379e5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9577d97 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 03:06:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 03:06:53 oak-gw06 kernel: LustreError: 11401:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880418abcc00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 03:06:53 oak-gw06 kernel: LustreError: 11401:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 03:06:53 oak-gw06 kernel: LustreError: 11401:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880418abcc00) refcount = 2 Aug 10 03:06:53 oak-gw06 kernel: LustreError: 11401:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 03:06:53 oak-gw06 kernel: LustreError: 11401:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803b5bce000/0xf077f1a82d8379e5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9577d97 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 03:06:53 oak-gw06 kernel: LustreError: 11401:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 03:12:00 oak-gw06 kernel: LustreError: 11415:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880136537240) refcount = 2 Aug 10 03:12:00 oak-gw06 kernel: LustreError: 11415:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 03:12:00 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 03:12:00 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 03:17:10 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 03:17:10 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 03:17:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502359930, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880271e26a00/0xf077f1a82d83db67 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb95871cc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 03:17:10 oak-gw06 kernel: LustreError: 11423:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88042b490f00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 03:17:10 oak-gw06 kernel: LustreError: 11423:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 03:17:10 oak-gw06 kernel: LustreError: 11423:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042b490f00) refcount = 2 Aug 10 03:17:10 oak-gw06 kernel: LustreError: 11423:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 03:17:10 oak-gw06 kernel: LustreError: 11423:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880271e26a00/0xf077f1a82d83db67 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb95871cc expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 03:17:10 oak-gw06 kernel: LustreError: 11423:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 03:17:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 03:22:20 oak-gw06 kernel: LustreError: 11481:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802d5705cc0) refcount = 2 Aug 10 03:22:20 oak-gw06 kernel: LustreError: 11481:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 03:22:20 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 03:22:20 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 03:27:30 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 03:27:30 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 03:27:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502360550, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88028d245c00/0xf077f1a82d864328 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb959677b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 03:27:30 oak-gw06 kernel: LustreError: 11578:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88033d2583c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 03:27:30 oak-gw06 kernel: LustreError: 11578:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 03:27:30 oak-gw06 kernel: LustreError: 11578:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88033d2583c0) refcount = 2 Aug 10 03:27:30 oak-gw06 kernel: LustreError: 11578:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 03:27:30 oak-gw06 kernel: LustreError: 11578:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88028d245c00/0xf077f1a82d864328 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb959677b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 03:27:30 oak-gw06 kernel: LustreError: 11578:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 03:27:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 03:32:37 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 03:32:37 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 03:37:44 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 03:37:44 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 03:37:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502361164, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802210c7c00/0xf077f1a82d930f36 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb95a5c0b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 03:37:44 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 03:37:44 oak-gw06 kernel: LustreError: 11657:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802e364f540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 03:37:44 oak-gw06 kernel: LustreError: 11657:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802e364f540) refcount = 2 Aug 10 03:37:44 oak-gw06 kernel: LustreError: 11657:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 03:37:44 oak-gw06 kernel: LustreError: 11657:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802210c7c00/0xf077f1a82d930f36 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb95a5c0b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 03:42:54 oak-gw06 kernel: LustreError: 11681:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88034d3d3840) refcount = 2 Aug 10 03:42:54 oak-gw06 kernel: LustreError: 11681:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 03:42:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 03:42:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 03:48:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 03:48:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 03:48:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502361782, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88021581ae00/0xf077f1a82d93e8e5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb95b5086 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 03:48:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 03:48:02 oak-gw06 kernel: LustreError: 11688:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88002accd840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 03:48:02 oak-gw06 kernel: LustreError: 11688:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 03:48:02 oak-gw06 kernel: LustreError: 11688:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88002accd840) refcount = 2 Aug 10 03:48:02 oak-gw06 kernel: LustreError: 11688:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 03:48:02 oak-gw06 kernel: LustreError: 11688:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88021581ae00/0xf077f1a82d93e8e5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb95b5086 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 03:48:02 oak-gw06 kernel: LustreError: 11688:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 03:53:12 oak-gw06 kernel: LustreError: 11777:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802f72cef00) refcount = 2 Aug 10 03:53:12 oak-gw06 kernel: LustreError: 11777:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 03:53:12 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 03:53:12 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 03:58:18 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 03:58:18 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 03:58:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502362398, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802a2745600/0xf077f1a82d9aac38 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb95c4475 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 03:58:18 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 03:58:18 oak-gw06 kernel: LustreError: 11862:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800203accc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 03:58:18 oak-gw06 kernel: LustreError: 11862:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 03:58:18 oak-gw06 kernel: LustreError: 11862:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800203accc0) refcount = 2 Aug 10 03:58:18 oak-gw06 kernel: LustreError: 11862:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 03:58:18 oak-gw06 kernel: LustreError: 11862:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802a2745600/0xf077f1a82d9aac38 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb95c4475 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 03:58:18 oak-gw06 kernel: LustreError: 11862:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 04:03:27 oak-gw06 kernel: LustreError: 11923:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880036cd0d80) refcount = 2 Aug 10 04:03:27 oak-gw06 kernel: LustreError: 11923:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 04:03:27 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 04:03:27 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 04:08:32 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 04:08:32 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 04:08:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502363012, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b28ca200/0xf077f1a82da239e3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb95d3809 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 04:08:32 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 04:08:32 oak-gw06 kernel: LustreError: 11977:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88031894bcc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 04:08:32 oak-gw06 kernel: LustreError: 11977:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 04:08:32 oak-gw06 kernel: LustreError: 11977:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88031894bcc0) refcount = 2 Aug 10 04:08:32 oak-gw06 kernel: LustreError: 11977:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 04:08:32 oak-gw06 kernel: LustreError: 11977:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b28ca200/0xf077f1a82da239e3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb95d3809 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 04:08:32 oak-gw06 kernel: LustreError: 11977:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 04:13:42 oak-gw06 kernel: LustreError: 12136:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802c67fe3c0) refcount = 2 Aug 10 04:13:42 oak-gw06 kernel: LustreError: 12136:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 04:13:42 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 04:13:42 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 04:18:53 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 04:18:53 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 04:18:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502363633, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800b58f2400/0xf077f1a82db2c5ca lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb95e2eec expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 04:18:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 04:18:53 oak-gw06 kernel: LustreError: 12231:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88002790c9c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 04:18:53 oak-gw06 kernel: LustreError: 12231:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 04:18:53 oak-gw06 kernel: LustreError: 12231:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88002790c9c0) refcount = 2 Aug 10 04:18:53 oak-gw06 kernel: LustreError: 12231:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 04:18:53 oak-gw06 kernel: LustreError: 12231:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800b58f2400/0xf077f1a82db2c5ca lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb95e2eec expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 04:18:53 oak-gw06 kernel: LustreError: 12231:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 04:24:01 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 04:24:01 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 04:29:07 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 04:29:07 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 04:29:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502364247, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880120e9f000/0xf077f1a82dbd9b1a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb95f223a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 04:29:07 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 04:29:07 oak-gw06 kernel: LustreError: 12404:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88007769e600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 04:29:07 oak-gw06 kernel: LustreError: 12404:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88007769e600) refcount = 2 Aug 10 04:29:07 oak-gw06 kernel: LustreError: 12404:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 04:29:07 oak-gw06 kernel: LustreError: 12404:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880120e9f000/0xf077f1a82dbd9b1a lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb95f223a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 04:34:13 oak-gw06 kernel: LustreError: 12490:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801384fd900) refcount = 2 Aug 10 04:34:13 oak-gw06 kernel: LustreError: 12490:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 04:34:13 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 04:34:13 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 04:39:22 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 04:39:22 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 04:39:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502364862, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802e08d6200/0xf077f1a82dcc7c1d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9601717 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 04:39:22 oak-gw06 kernel: LustreError: 12582:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88015a89c240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 04:39:22 oak-gw06 kernel: LustreError: 12582:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 04:39:22 oak-gw06 kernel: LustreError: 12582:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88015a89c240) refcount = 2 Aug 10 04:39:22 oak-gw06 kernel: LustreError: 12582:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 04:39:22 oak-gw06 kernel: LustreError: 12582:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802e08d6200/0xf077f1a82dcc7c1d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9601717 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 04:39:22 oak-gw06 kernel: LustreError: 12582:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 04:39:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 04:44:32 oak-gw06 kernel: LustreError: 12699:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803eaa45540) refcount = 2 Aug 10 04:44:32 oak-gw06 kernel: LustreError: 12699:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 04:44:32 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 04:44:32 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 04:49:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 04:49:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 04:49:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502365482, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88014b651c00/0xf077f1a82ddc1dd8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9610ce2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 04:49:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 04:49:42 oak-gw06 kernel: LustreError: 12859:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88014e2bd540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 04:49:42 oak-gw06 kernel: LustreError: 12859:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 04:49:42 oak-gw06 kernel: LustreError: 12859:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88014e2bd540) refcount = 2 Aug 10 04:49:42 oak-gw06 kernel: LustreError: 12859:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 04:49:42 oak-gw06 kernel: LustreError: 12859:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88014b651c00/0xf077f1a82ddc1dd8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9610ce2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 04:49:42 oak-gw06 kernel: LustreError: 12859:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 04:54:50 oak-gw06 kernel: LustreError: 12882:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88010bc05c00) refcount = 2 Aug 10 04:54:50 oak-gw06 kernel: LustreError: 12882:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 04:54:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 04:54:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 04:59:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 04:59:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 04:59:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502366099, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801c21c9200/0xf077f1a82de9628a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9620298 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 04:59:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 04:59:59 oak-gw06 kernel: LustreError: 12897:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88012e9f36c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 04:59:59 oak-gw06 kernel: LustreError: 12897:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 04:59:59 oak-gw06 kernel: LustreError: 12897:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88012e9f36c0) refcount = 2 Aug 10 04:59:59 oak-gw06 kernel: LustreError: 12897:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 04:59:59 oak-gw06 kernel: LustreError: 12897:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801c21c9200/0xf077f1a82de9628a lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9620298 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 04:59:59 oak-gw06 kernel: LustreError: 12897:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 05:05:10 oak-gw06 kernel: LustreError: 12944:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880145b80900) refcount = 2 Aug 10 05:05:10 oak-gw06 kernel: LustreError: 12944:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 05:05:10 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 05:05:10 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 05:10:20 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 05:10:20 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 05:10:20 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502366720, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801343b2c00/0xf077f1a82dea9b37 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb962f847 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 05:10:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 05:10:21 oak-gw06 kernel: LustreError: 12957:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88034b7fb600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 05:10:21 oak-gw06 kernel: LustreError: 12957:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 05:10:21 oak-gw06 kernel: LustreError: 12957:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88034b7fb600) refcount = 2 Aug 10 05:10:21 oak-gw06 kernel: LustreError: 12957:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 05:10:21 oak-gw06 kernel: LustreError: 12957:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801343b2c00/0xf077f1a82dea9b37 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb962f847 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 05:10:21 oak-gw06 kernel: LustreError: 12957:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 05:15:32 oak-gw06 kernel: LustreError: 12966:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88008be4a6c0) refcount = 2 Aug 10 05:15:32 oak-gw06 kernel: LustreError: 12966:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 05:15:32 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 05:15:32 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 05:20:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 05:20:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 05:20:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502367342, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880171f5f000/0xf077f1a82deaef06 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb963ed6a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 05:20:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 05:20:42 oak-gw06 kernel: LustreError: 12977:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802dca2e000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 05:20:42 oak-gw06 kernel: LustreError: 12977:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 05:20:42 oak-gw06 kernel: LustreError: 12977:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802dca2e000) refcount = 2 Aug 10 05:20:42 oak-gw06 kernel: LustreError: 12977:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 05:20:42 oak-gw06 kernel: LustreError: 12977:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880171f5f000/0xf077f1a82deaef06 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb963ed6a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 05:20:42 oak-gw06 kernel: LustreError: 12977:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 05:25:49 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 05:25:49 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 05:30:57 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 05:30:57 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 05:30:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502367957, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802f4c1e600/0xf077f1a82deb7819 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb964e240 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 05:30:57 oak-gw06 kernel: LustreError: 12999:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880018916900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 05:30:57 oak-gw06 kernel: LustreError: 12999:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880018916900) refcount = 2 Aug 10 05:30:57 oak-gw06 kernel: LustreError: 12999:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 05:30:57 oak-gw06 kernel: LustreError: 12999:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802f4c1e600/0xf077f1a82deb7819 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb964e240 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 05:30:57 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 05:36:07 oak-gw06 kernel: LustreError: 13007:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880059aea480) refcount = 2 Aug 10 05:36:07 oak-gw06 kernel: LustreError: 13007:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 05:36:07 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 05:36:07 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 05:41:17 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 05:41:17 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 05:41:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502368577, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880027aaa600/0xf077f1a82dec275f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb965d5e2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 05:41:17 oak-gw06 kernel: LustreError: 13028:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880194906780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 05:41:17 oak-gw06 kernel: LustreError: 13028:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 05:41:17 oak-gw06 kernel: LustreError: 13028:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880194906780) refcount = 2 Aug 10 05:41:17 oak-gw06 kernel: LustreError: 13028:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 05:41:17 oak-gw06 kernel: LustreError: 13028:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880027aaa600/0xf077f1a82dec275f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb965d5e2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 05:41:17 oak-gw06 kernel: LustreError: 13028:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 05:41:17 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 05:46:30 oak-gw06 kernel: LustreError: 13032:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801847da540) refcount = 2 Aug 10 05:46:30 oak-gw06 kernel: LustreError: 13032:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 05:46:30 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 05:46:30 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 05:51:37 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 05:51:37 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 05:51:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502369197, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880110a34000/0xf077f1a82decb3c1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb966cc01 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 05:51:37 oak-gw06 kernel: LustreError: 13047:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880139e25d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 05:51:37 oak-gw06 kernel: LustreError: 13047:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 05:51:37 oak-gw06 kernel: LustreError: 13047:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880139e25d80) refcount = 2 Aug 10 05:51:37 oak-gw06 kernel: LustreError: 13047:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 05:51:37 oak-gw06 kernel: LustreError: 13047:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880110a34000/0xf077f1a82decb3c1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb966cc01 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 05:51:37 oak-gw06 kernel: LustreError: 13047:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 05:51:37 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 05:56:43 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 05:56:43 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 06:01:51 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 06:01:51 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 06:01:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502369811, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88018aa24800/0xf077f1a82ded00e4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb967be14 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 06:01:51 oak-gw06 kernel: LustreError: 13094:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803075ec300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 06:01:51 oak-gw06 kernel: LustreError: 13094:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803075ec300) refcount = 2 Aug 10 06:01:51 oak-gw06 kernel: LustreError: 13094:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 06:01:51 oak-gw06 kernel: LustreError: 13094:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88018aa24800/0xf077f1a82ded00e4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb967be14 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 06:01:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 06:07:02 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 06:07:02 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 06:12:08 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 06:12:08 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 06:12:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502370428, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802340ef600/0xf077f1a82ded4f5e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb968b34c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 06:12:09 oak-gw06 kernel: LustreError: 13113:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88022d4f1c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 06:12:09 oak-gw06 kernel: LustreError: 13113:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88022d4f1c00) refcount = 2 Aug 10 06:12:09 oak-gw06 kernel: LustreError: 13113:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 06:12:09 oak-gw06 kernel: LustreError: 13113:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802340ef600/0xf077f1a82ded4f5e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb968b34c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 06:12:09 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 06:17:15 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 06:17:15 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 06:22:23 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 06:22:23 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 06:22:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502371043, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880327470800/0xf077f1a82def11f7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb969a6d2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 06:22:23 oak-gw06 kernel: LustreError: 13252:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880426a00a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 06:22:23 oak-gw06 kernel: LustreError: 13252:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880426a00a80) refcount = 2 Aug 10 06:22:23 oak-gw06 kernel: LustreError: 13252:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 06:22:23 oak-gw06 kernel: LustreError: 13252:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880327470800/0xf077f1a82def11f7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb969a6d2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 06:22:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 06:27:31 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 06:27:31 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 06:32:36 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 06:32:36 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 06:32:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502371656, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88023cabc600/0xf077f1a82dfefa37 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb96a9939 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 06:32:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 06:32:36 oak-gw06 kernel: LustreError: 13391:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880130311480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 06:32:36 oak-gw06 kernel: LustreError: 13391:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880130311480) refcount = 2 Aug 10 06:32:36 oak-gw06 kernel: LustreError: 13391:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 06:32:36 oak-gw06 kernel: LustreError: 13391:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88023cabc600/0xf077f1a82dfefa37 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb96a9939 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 06:37:43 oak-gw06 kernel: LustreError: 13400:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801204ef9c0) refcount = 2 Aug 10 06:37:43 oak-gw06 kernel: LustreError: 13400:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 06:37:43 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 06:37:43 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 06:42:51 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 06:42:51 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 06:42:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502372271, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88027a369c00/0xf077f1a82e01a1b5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb96b8dbb expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 06:42:51 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 06:42:51 oak-gw06 kernel: LustreError: 13410:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880059aea000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 06:42:51 oak-gw06 kernel: LustreError: 13410:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 06:42:51 oak-gw06 kernel: LustreError: 13410:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880059aea000) refcount = 2 Aug 10 06:42:51 oak-gw06 kernel: LustreError: 13410:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 06:42:51 oak-gw06 kernel: LustreError: 13410:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88027a369c00/0xf077f1a82e01a1b5 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb96b8dbb expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 06:42:51 oak-gw06 kernel: LustreError: 13410:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 06:47:57 oak-gw06 kernel: LustreError: 13419:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880131ba3780) refcount = 2 Aug 10 06:47:57 oak-gw06 kernel: LustreError: 13419:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 06:47:57 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 06:47:57 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 06:53:08 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 06:53:08 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 06:53:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502372888, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88022fb20000/0xf077f1a82e01ffa9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb96c8244 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 06:53:08 oak-gw06 kernel: LustreError: 13433:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801f9f11840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 06:53:08 oak-gw06 kernel: LustreError: 13433:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 06:53:08 oak-gw06 kernel: LustreError: 13433:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801f9f11840) refcount = 2 Aug 10 06:53:08 oak-gw06 kernel: LustreError: 13433:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 06:53:08 oak-gw06 kernel: LustreError: 13433:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88022fb20000/0xf077f1a82e01ffa9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb96c8244 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 06:53:08 oak-gw06 kernel: LustreError: 13433:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 06:53:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 06:58:17 oak-gw06 kernel: LustreError: 13441:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88010bdf2300) refcount = 2 Aug 10 06:58:17 oak-gw06 kernel: LustreError: 13441:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 06:58:17 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 06:58:17 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 07:03:28 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 07:03:28 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 07:03:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502373508, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880013fa0c00/0xf077f1a82e02a342 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb96d7966 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 07:03:28 oak-gw06 kernel: LustreError: 13484:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880007b8d300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 07:03:28 oak-gw06 kernel: LustreError: 13484:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 07:03:28 oak-gw06 kernel: LustreError: 13484:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880007b8d300) refcount = 2 Aug 10 07:03:28 oak-gw06 kernel: LustreError: 13484:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 07:03:28 oak-gw06 kernel: LustreError: 13484:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880013fa0c00/0xf077f1a82e02a342 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb96d7966 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 07:03:28 oak-gw06 kernel: LustreError: 13484:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 07:03:28 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 07:08:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 07:08:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 07:13:41 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 07:13:41 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 07:13:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502374121, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880136c93e00/0xf077f1a82e030cc7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb96e6f15 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 07:13:41 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 07:13:41 oak-gw06 kernel: LustreError: 13508:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802f63b0900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 07:13:41 oak-gw06 kernel: LustreError: 13508:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802f63b0900) refcount = 2 Aug 10 07:13:41 oak-gw06 kernel: LustreError: 13508:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 07:13:41 oak-gw06 kernel: LustreError: 13508:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880136c93e00/0xf077f1a82e030cc7 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb96e6f15 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 07:18:52 oak-gw06 kernel: LustreError: 13512:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880068a24f00) refcount = 2 Aug 10 07:18:52 oak-gw06 kernel: LustreError: 13512:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 07:18:52 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 07:18:52 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 07:23:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 07:23:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 07:23:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502374739, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880120b25600/0xf077f1a82e035e20 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb96f6382 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 07:23:59 oak-gw06 kernel: LustreError: 13525:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880068a24e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 07:23:59 oak-gw06 kernel: LustreError: 13525:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 07:23:59 oak-gw06 kernel: LustreError: 13525:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880068a24e40) refcount = 2 Aug 10 07:23:59 oak-gw06 kernel: LustreError: 13525:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 07:23:59 oak-gw06 kernel: LustreError: 13525:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880120b25600/0xf077f1a82e035e20 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb96f6382 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 07:23:59 oak-gw06 kernel: LustreError: 13525:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 07:23:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 07:29:05 oak-gw06 kernel: LustreError: 13529:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802f72ce900) refcount = 2 Aug 10 07:29:05 oak-gw06 kernel: LustreError: 13529:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 07:29:05 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 07:29:05 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 07:34:11 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 07:34:11 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 07:34:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502375351, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880120b24800/0xf077f1a82e038d52 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb97057a9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 07:34:11 oak-gw06 kernel: LustreError: 13539:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802f72ce3c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 07:34:11 oak-gw06 kernel: LustreError: 13539:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 07:34:11 oak-gw06 kernel: LustreError: 13539:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802f72ce3c0) refcount = 2 Aug 10 07:34:11 oak-gw06 kernel: LustreError: 13539:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 07:34:11 oak-gw06 kernel: LustreError: 13539:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880120b24800/0xf077f1a82e038d52 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb97057a9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 07:34:11 oak-gw06 kernel: LustreError: 13539:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 07:34:11 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 07:39:20 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 07:39:20 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 07:44:30 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 07:44:30 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 07:44:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502375970, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880123bb4800/0xf077f1a82e03c6e1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9714d0b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 07:44:30 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 07:44:30 oak-gw06 kernel: LustreError: 13557:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880399dcaf00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 07:44:30 oak-gw06 kernel: LustreError: 13557:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880399dcaf00) refcount = 2 Aug 10 07:44:30 oak-gw06 kernel: LustreError: 13557:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 07:44:30 oak-gw06 kernel: LustreError: 13557:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880123bb4800/0xf077f1a82e03c6e1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9714d0b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 07:49:39 oak-gw06 kernel: LustreError: 13561:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c8f7df00) refcount = 2 Aug 10 07:49:39 oak-gw06 kernel: LustreError: 13561:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 07:49:39 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 07:49:39 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 07:54:47 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 07:54:47 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 07:54:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502376587, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880287ab5a00/0xf077f1a82e03f621 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9724212 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 07:54:47 oak-gw06 kernel: LustreError: 13573:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800387f7000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 07:54:47 oak-gw06 kernel: LustreError: 13573:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 07:54:47 oak-gw06 kernel: LustreError: 13573:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800387f7000) refcount = 2 Aug 10 07:54:47 oak-gw06 kernel: LustreError: 13573:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 07:54:47 oak-gw06 kernel: LustreError: 13573:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880287ab5a00/0xf077f1a82e03f621 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9724212 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 07:54:47 oak-gw06 kernel: LustreError: 13573:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 07:54:47 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 07:59:58 oak-gw06 kernel: LustreError: 13581:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88041f7c2780) refcount = 2 Aug 10 07:59:58 oak-gw06 kernel: LustreError: 13581:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 07:59:58 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 07:59:58 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 08:05:03 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 08:05:03 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 08:05:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502377203, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800ac13a000/0xf077f1a82e043d9b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb97334e2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 08:05:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 08:05:03 oak-gw06 kernel: LustreError: 13626:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88008a83bb40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 08:05:03 oak-gw06 kernel: LustreError: 13626:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 08:05:03 oak-gw06 kernel: LustreError: 13626:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88008a83bb40) refcount = 2 Aug 10 08:05:03 oak-gw06 kernel: LustreError: 13626:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 08:05:03 oak-gw06 kernel: LustreError: 13626:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800ac13a000/0xf077f1a82e043d9b lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb97334e2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 08:05:03 oak-gw06 kernel: LustreError: 13626:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 08:10:12 oak-gw06 kernel: LustreError: 13642:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88010debcb40) refcount = 2 Aug 10 08:10:12 oak-gw06 kernel: LustreError: 13642:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 08:10:12 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 08:10:12 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 08:15:22 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 08:15:22 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 08:15:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502377822, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880109e42200/0xf077f1a82e04a13f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9742638 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 08:15:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 08:15:22 oak-gw06 kernel: LustreError: 13645:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880195b45840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 08:15:22 oak-gw06 kernel: LustreError: 13645:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 08:15:22 oak-gw06 kernel: LustreError: 13645:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880195b45840) refcount = 2 Aug 10 08:15:22 oak-gw06 kernel: LustreError: 13645:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 08:15:22 oak-gw06 kernel: LustreError: 13645:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880109e42200/0xf077f1a82e04a13f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9742638 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 08:15:22 oak-gw06 kernel: LustreError: 13645:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 08:20:30 oak-gw06 kernel: LustreError: 13660:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880195b450c0) refcount = 2 Aug 10 08:20:30 oak-gw06 kernel: LustreError: 13660:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 08:20:30 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 08:20:30 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 08:25:35 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 08:25:35 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 08:25:35 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502378435, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880400e16600/0xf077f1a82e04da7a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb975191d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 08:25:35 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 08:25:35 oak-gw06 kernel: LustreError: 13665:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880046d42c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 08:25:35 oak-gw06 kernel: LustreError: 13665:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 08:25:35 oak-gw06 kernel: LustreError: 13665:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880046d42c00) refcount = 2 Aug 10 08:25:35 oak-gw06 kernel: LustreError: 13665:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 08:25:35 oak-gw06 kernel: LustreError: 13665:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880400e16600/0xf077f1a82e04da7a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb975191d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 08:25:35 oak-gw06 kernel: LustreError: 13665:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 08:30:44 oak-gw06 kernel: LustreError: 13677:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88010bdbcd80) refcount = 1 Aug 10 08:30:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 08:30:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 08:35:49 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 08:35:49 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 08:35:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502379049, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88013d946e00/0xf077f1a82e04f9c9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9760aea expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 08:35:49 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 08:35:49 oak-gw06 kernel: LustreError: 13679:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88034e6a06c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 08:35:49 oak-gw06 kernel: LustreError: 13679:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 08:35:49 oak-gw06 kernel: LustreError: 13679:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88034e6a06c0) refcount = 2 Aug 10 08:35:49 oak-gw06 kernel: LustreError: 13679:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 08:35:49 oak-gw06 kernel: LustreError: 13679:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88013d946e00/0xf077f1a82e04f9c9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9760aea expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 08:40:54 oak-gw06 kernel: LustreError: 13691:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ac6d1780) refcount = 2 Aug 10 08:40:54 oak-gw06 kernel: LustreError: 13691:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 08:40:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 08:40:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 08:46:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 08:46:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 08:46:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502379661, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801a2661600/0xf077f1a82e050b18 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb976fc78 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 08:46:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 08:46:02 oak-gw06 kernel: LustreError: 13699:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ac6d1480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 08:46:02 oak-gw06 kernel: LustreError: 13699:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 08:46:02 oak-gw06 kernel: LustreError: 13699:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ac6d1480) refcount = 2 Aug 10 08:46:02 oak-gw06 kernel: LustreError: 13699:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 08:46:02 oak-gw06 kernel: LustreError: 13699:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801a2661600/0xf077f1a82e050b18 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb976fc78 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 08:46:02 oak-gw06 kernel: LustreError: 13699:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 08:51:11 oak-gw06 kernel: LustreError: 13714:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ac6d1480) refcount = 2 Aug 10 08:51:11 oak-gw06 kernel: LustreError: 13714:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 08:51:11 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 08:51:11 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 08:56:21 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 08:56:21 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 08:56:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502380281, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801a15fde00/0xf077f1a82e057163 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb977efcd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 08:56:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 08:56:21 oak-gw06 kernel: LustreError: 13719:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88011a4da300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 08:56:21 oak-gw06 kernel: LustreError: 13719:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 08:56:21 oak-gw06 kernel: LustreError: 13719:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88011a4da300) refcount = 2 Aug 10 08:56:21 oak-gw06 kernel: LustreError: 13719:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 08:56:21 oak-gw06 kernel: LustreError: 13719:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801a15fde00/0xf077f1a82e057163 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb977efcd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 08:56:21 oak-gw06 kernel: LustreError: 13719:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 09:01:30 oak-gw06 kernel: LustreError: 13767:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88019235ab40) refcount = 2 Aug 10 09:01:30 oak-gw06 kernel: LustreError: 13767:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 09:01:30 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 09:01:30 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 09:06:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 09:06:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 09:06:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502380902, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88028d26e800/0xf077f1a82e05c44b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb978e1f5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 09:06:42 oak-gw06 kernel: LustreError: 13770:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801ffdfb9c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 09:06:42 oak-gw06 kernel: LustreError: 13770:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 09:06:42 oak-gw06 kernel: LustreError: 13770:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ffdfb9c0) refcount = 2 Aug 10 09:06:42 oak-gw06 kernel: LustreError: 13770:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 09:06:42 oak-gw06 kernel: LustreError: 13770:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88028d26e800/0xf077f1a82e05c44b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb978e1f5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 09:06:42 oak-gw06 kernel: LustreError: 13770:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 09:06:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 09:11:55 oak-gw06 kernel: LustreError: 13784:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880139ecfd80) refcount = 2 Aug 10 09:11:55 oak-gw06 kernel: LustreError: 13784:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 09:11:55 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 09:11:55 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 09:17:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 09:17:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 09:17:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502381520, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880021397200/0xf077f1a82e062ff9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb979d5d6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 09:17:00 oak-gw06 kernel: LustreError: 13791:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880269b833c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 09:17:00 oak-gw06 kernel: LustreError: 13791:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 09:17:00 oak-gw06 kernel: LustreError: 13791:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880269b833c0) refcount = 2 Aug 10 09:17:00 oak-gw06 kernel: LustreError: 13791:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 09:17:00 oak-gw06 kernel: LustreError: 13791:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880021397200/0xf077f1a82e062ff9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb979d5d6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 09:17:00 oak-gw06 kernel: LustreError: 13791:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 09:17:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 09:22:09 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 09:22:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 09:27:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 09:27:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 09:27:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502382136, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88022fb23a00/0xf077f1a82e06984a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb97ac72c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 09:27:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 09:32:28 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 09:32:28 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 09:37:34 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 09:37:34 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 09:37:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502382754, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802b28c8e00/0xf077f1a82e06c1a2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb97bb897 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 09:37:34 oak-gw06 kernel: LustreError: 13823:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880393ba8e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 09:37:34 oak-gw06 kernel: LustreError: 13823:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880393ba8e40) refcount = 2 Aug 10 09:37:34 oak-gw06 kernel: LustreError: 13823:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 09:37:34 oak-gw06 kernel: LustreError: 13823:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802b28c8e00/0xf077f1a82e06c1a2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb97bb897 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 09:37:34 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 09:42:46 oak-gw06 kernel: LustreError: 13836:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88027b307600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 09:42:46 oak-gw06 kernel: LustreError: 13836:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88027b307600) refcount = 2 Aug 10 09:42:46 oak-gw06 kernel: LustreError: 13836:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 09:42:46 oak-gw06 kernel: LustreError: 13836:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880110f91800/0xf077f1a82e06de58 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb97c2d59 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 09:42:46 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 09:42:46 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 09:47:59 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 09:47:59 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 09:47:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502383379, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880356334600/0xf077f1a82e06ff52 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb97caaf7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 09:47:59 oak-gw06 kernel: LustreError: 13844:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88008a83b600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 09:47:59 oak-gw06 kernel: LustreError: 13844:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88008a83b600) refcount = 2 Aug 10 09:47:59 oak-gw06 kernel: LustreError: 13844:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 09:47:59 oak-gw06 kernel: LustreError: 13844:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880356334600/0xf077f1a82e06ff52 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb97caaf7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 09:47:59 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 09:53:05 oak-gw06 kernel: LustreError: 13855:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88010060f240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 09:53:05 oak-gw06 kernel: LustreError: 13855:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88010060f240) refcount = 2 Aug 10 09:53:05 oak-gw06 kernel: LustreError: 13855:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 09:53:05 oak-gw06 kernel: LustreError: 13855:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88004d695800/0xf077f1a82e0713f7 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb97d1ef5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 09:53:05 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 09:53:05 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 09:58:13 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 09:58:13 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 09:58:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502383993, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88004d697c00/0xf077f1a82e072dab lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb97d9c31 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 09:58:13 oak-gw06 kernel: LustreError: 13859:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803d4afb0c0) refcount = 2 Aug 10 09:58:13 oak-gw06 kernel: LustreError: 13859:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 09:58:13 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 10:03:20 oak-gw06 kernel: LustreError: 13906:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802bb641600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 10:03:20 oak-gw06 kernel: LustreError: 13906:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 10:03:20 oak-gw06 kernel: LustreError: 13906:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bb641600) refcount = 2 Aug 10 10:03:20 oak-gw06 kernel: LustreError: 13906:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 10:03:20 oak-gw06 kernel: LustreError: 13906:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803d6704000/0xf077f1a82e074b6b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb97e0fc6 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 10:03:20 oak-gw06 kernel: LustreError: 13906:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 10:03:20 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 10:03:20 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 10:08:33 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 10:08:33 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 10:08:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502384613, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8800a163ce00/0xf077f1a82e076df4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb97e8d79 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 10:08:33 oak-gw06 kernel: LustreError: 13911:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880154b026c0) refcount = 2 Aug 10 10:08:33 oak-gw06 kernel: LustreError: 13911:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 10:08:33 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 10:13:42 oak-gw06 kernel: LustreError: 13923:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803ced783c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 10:13:42 oak-gw06 kernel: LustreError: 13923:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 10:13:42 oak-gw06 kernel: LustreError: 13923:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ced783c0) refcount = 2 Aug 10 10:13:42 oak-gw06 kernel: LustreError: 13923:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 10:13:42 oak-gw06 kernel: LustreError: 13923:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801107cb200/0xf077f1a82e079a39 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb97f03a7 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 10:13:42 oak-gw06 kernel: LustreError: 13923:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 10:13:42 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 10:13:42 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 10:18:50 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 10:18:50 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 10:18:50 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502385230, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801107c9a00/0xf077f1a82e07b250 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb97f8161 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 10:18:50 oak-gw06 kernel: LustreError: 13931:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bdf9f540) refcount = 2 Aug 10 10:18:50 oak-gw06 kernel: LustreError: 13931:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 10:18:50 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 10:23:57 oak-gw06 kernel: LustreError: 13943:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880279ba0e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 10:23:57 oak-gw06 kernel: LustreError: 13943:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 10:23:57 oak-gw06 kernel: LustreError: 13943:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880279ba0e40) refcount = 2 Aug 10 10:23:57 oak-gw06 kernel: LustreError: 13943:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 10:23:57 oak-gw06 kernel: LustreError: 13943:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880159277200/0xf077f1a82e07c32f lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb97ff55f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 10:23:57 oak-gw06 kernel: LustreError: 13943:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 10:23:57 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 10:23:57 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 10:29:06 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 10:29:06 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 10:29:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502385846, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880159277a00/0xf077f1a82e07d18a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9807255 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 10:29:06 oak-gw06 kernel: LustreError: 13947:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88014ef70600) refcount = 2 Aug 10 10:29:06 oak-gw06 kernel: LustreError: 13947:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 10:29:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 10:34:18 oak-gw06 kernel: LustreError: 13958:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8803051dae40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 10:34:18 oak-gw06 kernel: LustreError: 13958:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 10:34:18 oak-gw06 kernel: LustreError: 13958:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803051dae40) refcount = 2 Aug 10 10:34:18 oak-gw06 kernel: LustreError: 13958:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 10:34:18 oak-gw06 kernel: LustreError: 13958:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880110c8a000/0xf077f1a82e07eb99 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb980e9c5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 10:34:18 oak-gw06 kernel: LustreError: 13958:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 10:34:18 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 10:34:18 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 10:39:27 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 10:39:27 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 10:39:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502386467, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880123bb5400/0xf077f1a82e08004c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb98165fe expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 10:39:27 oak-gw06 kernel: LustreError: 13961:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042c4bf6c0) refcount = 2 Aug 10 10:39:27 oak-gw06 kernel: LustreError: 13961:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 10:39:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 10:44:40 oak-gw06 kernel: LustreError: 13972:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88008a83be40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 10:44:40 oak-gw06 kernel: LustreError: 13972:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 10:44:40 oak-gw06 kernel: LustreError: 13972:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88008a83be40) refcount = 2 Aug 10 10:44:40 oak-gw06 kernel: LustreError: 13972:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 10:44:40 oak-gw06 kernel: LustreError: 13972:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803605d6c00/0xf077f1a82e0811cc lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb981dc25 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 10:44:40 oak-gw06 kernel: LustreError: 13972:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 10:44:40 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 10:44:40 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 10:49:53 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 10:49:53 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 10:49:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502387093, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801b7afc000/0xf077f1a82e0811e8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9825a41 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 10:49:53 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 10:49:53 oak-gw06 kernel: LustreError: 13980:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802f63b0540) refcount = 2 Aug 10 10:49:53 oak-gw06 kernel: LustreError: 13980:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 10:55:00 oak-gw06 kernel: LustreError: 13996:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802f63b0540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 10:55:00 oak-gw06 kernel: LustreError: 13996:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 10:55:00 oak-gw06 kernel: LustreError: 13996:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802f63b0540) refcount = 2 Aug 10 10:55:00 oak-gw06 kernel: LustreError: 13996:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 10:55:00 oak-gw06 kernel: LustreError: 13996:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801b7afe200/0xf077f1a82e081b42 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb982d141 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 10:55:00 oak-gw06 kernel: LustreError: 13996:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 10:55:00 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 10:55:00 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 11:00:06 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 11:00:06 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 11:00:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502387706, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801491e5c00/0xf077f1a82e08539d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9834c07 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 11:00:06 oak-gw06 kernel: LustreError: 14010:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801ffdfb600) refcount = 2 Aug 10 11:00:06 oak-gw06 kernel: LustreError: 14010:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 11:00:06 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 11:05:13 oak-gw06 kernel: LustreError: 14050:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880413f39d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 11:05:13 oak-gw06 kernel: LustreError: 14050:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 11:05:13 oak-gw06 kernel: LustreError: 14050:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413f39d80) refcount = 2 Aug 10 11:05:13 oak-gw06 kernel: LustreError: 14050:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 11:05:13 oak-gw06 kernel: LustreError: 14050:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880287e37000/0xf077f1a82e08af68 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb983c219 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 11:05:13 oak-gw06 kernel: LustreError: 14050:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 11:05:13 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 11:05:13 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 11:10:22 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 11:10:22 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 11:10:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502388322, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801d3a0b800/0xf077f1a82e0914f6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9843fc5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 11:10:22 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 11:10:22 oak-gw06 kernel: LustreError: 14065:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413f39780) refcount = 2 Aug 10 11:10:22 oak-gw06 kernel: LustreError: 14065:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 11:15:31 oak-gw06 kernel: LustreError: 14073:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802057a26c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 11:15:31 oak-gw06 kernel: LustreError: 14073:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 11:15:31 oak-gw06 kernel: LustreError: 14073:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802057a26c0) refcount = 2 Aug 10 11:15:31 oak-gw06 kernel: LustreError: 14073:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 11:15:31 oak-gw06 kernel: LustreError: 14073:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88011e839800/0xf077f1a82e0976fd lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb984b6a9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 11:15:31 oak-gw06 kernel: LustreError: 14073:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 11:15:31 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 11:15:31 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 11:20:40 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 11:20:40 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 11:20:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502388940, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88034312e600/0xf077f1a82e09ca2b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9853463 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 11:20:40 oak-gw06 kernel: LustreError: 14089:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803075ec9c0) refcount = 2 Aug 10 11:20:40 oak-gw06 kernel: LustreError: 14089:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 11:20:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 11:25:50 oak-gw06 kernel: LustreError: 14098:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802bb225000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 11:25:50 oak-gw06 kernel: LustreError: 14098:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 11:25:50 oak-gw06 kernel: LustreError: 14098:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802bb225000) refcount = 2 Aug 10 11:25:50 oak-gw06 kernel: LustreError: 14098:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 11:25:50 oak-gw06 kernel: LustreError: 14098:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88012edbf200/0xf077f1a82e0a1350 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb985aabb expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 11:25:50 oak-gw06 kernel: LustreError: 14098:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 11:25:50 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 11:25:50 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 11:30:58 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 11:30:58 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 11:30:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502389558, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88010155b400/0xf077f1a82e0a7462 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9862844 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 11:30:58 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 11:30:58 oak-gw06 kernel: LustreError: 14114:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88011950a240) refcount = 2 Aug 10 11:30:58 oak-gw06 kernel: LustreError: 14114:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 11:36:06 oak-gw06 kernel: LustreError: 14122:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880235dcf600) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 11:36:06 oak-gw06 kernel: LustreError: 14122:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 11:36:06 oak-gw06 kernel: LustreError: 14122:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880235dcf600) refcount = 2 Aug 10 11:36:06 oak-gw06 kernel: LustreError: 14122:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 11:36:06 oak-gw06 kernel: LustreError: 14122:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801a93c7e00/0xf077f1a82e0abcfb lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9869cd5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 11:36:06 oak-gw06 kernel: LustreError: 14122:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 11:36:06 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 11:36:06 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 11:41:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 11:41:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 11:41:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502390176, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880159275400/0xf077f1a82e0b0c55 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9871a65 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 11:41:16 oak-gw06 kernel: LustreError: 14138:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880279ba0780) refcount = 2 Aug 10 11:41:16 oak-gw06 kernel: LustreError: 14138:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 11:41:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 11:46:27 oak-gw06 kernel: LustreError: 14146:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8804289a6c00) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 11:46:27 oak-gw06 kernel: LustreError: 14146:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 11:46:27 oak-gw06 kernel: LustreError: 14146:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8804289a6c00) refcount = 2 Aug 10 11:46:27 oak-gw06 kernel: LustreError: 14146:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 11:46:27 oak-gw06 kernel: LustreError: 14146:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88041db99e00/0xf077f1a82e0b5fad lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb98790bd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 11:46:27 oak-gw06 kernel: LustreError: 14146:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 11:46:27 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 11:46:27 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 11:51:36 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 11:51:36 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 11:51:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502390796, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88013ed9f000/0xf077f1a82e0bc152 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9880f42 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 11:51:36 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 11:51:36 oak-gw06 kernel: LustreError: 14161:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803ec740300) refcount = 2 Aug 10 11:51:36 oak-gw06 kernel: LustreError: 14161:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 11:56:44 oak-gw06 kernel: LustreError: 14177:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801264a5b40) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 11:56:44 oak-gw06 kernel: LustreError: 14177:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 11:56:44 oak-gw06 kernel: LustreError: 14177:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801264a5b40) refcount = 2 Aug 10 11:56:44 oak-gw06 kernel: LustreError: 14177:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 11:56:44 oak-gw06 kernel: LustreError: 14177:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88034d3afa00/0xf077f1a82e0c18fc lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9888466 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 11:56:44 oak-gw06 kernel: LustreError: 14177:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 11:56:44 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 11:56:44 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 12:01:50 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 12:01:50 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 12:01:50 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502391410, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801bcf79200/0xf077f1a82e0d179c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb98900ad expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 12:01:50 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 12:01:50 oak-gw06 kernel: LustreError: 14224:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88005e02bc00) refcount = 2 Aug 10 12:01:50 oak-gw06 kernel: LustreError: 14224:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 12:06:58 oak-gw06 kernel: LustreError: 14229:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8800a73d5a80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 12:06:58 oak-gw06 kernel: LustreError: 14229:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 12:06:58 oak-gw06 kernel: LustreError: 14229:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8800a73d5a80) refcount = 2 Aug 10 12:06:58 oak-gw06 kernel: LustreError: 14229:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 12:06:58 oak-gw06 kernel: LustreError: 14229:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880147a43e00/0xf077f1a82e0d3850 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb989766b expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 12:06:58 oak-gw06 kernel: LustreError: 14229:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 12:06:58 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 12:06:58 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 12:12:08 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 12:12:08 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 12:12:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502392028, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880129943800/0xf077f1a82e0d5f9b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb989f505 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 12:12:08 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 12:12:08 oak-gw06 kernel: LustreError: 14243:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88014e2330c0) refcount = 2 Aug 10 12:12:08 oak-gw06 kernel: LustreError: 14243:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 12:17:18 oak-gw06 kernel: LustreError: 14248:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880407f1d480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 12:17:18 oak-gw06 kernel: LustreError: 14248:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 12:17:18 oak-gw06 kernel: LustreError: 14248:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880407f1d480) refcount = 2 Aug 10 12:17:18 oak-gw06 kernel: LustreError: 14248:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 12:17:18 oak-gw06 kernel: LustreError: 14248:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880129942a00/0xf077f1a82e0da4ec lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb98a6c83 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 12:17:18 oak-gw06 kernel: LustreError: 14248:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 12:17:18 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 12:17:18 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 12:22:27 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 12:22:27 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 12:22:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502392647, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88005a103800/0xf077f1a82e0dc695 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb98aea05 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 12:22:27 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 12:22:27 oak-gw06 kernel: LustreError: 14265:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801025193c0) refcount = 2 Aug 10 12:22:27 oak-gw06 kernel: LustreError: 14265:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 12:27:39 oak-gw06 kernel: LustreError: 14269:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801c5a38180) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 12:27:39 oak-gw06 kernel: LustreError: 14269:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 12:27:39 oak-gw06 kernel: LustreError: 14269:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c5a38180) refcount = 2 Aug 10 12:27:39 oak-gw06 kernel: LustreError: 14269:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 12:27:39 oak-gw06 kernel: LustreError: 14269:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802c210ea00/0xf077f1a82e0e172a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb98b5fca expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 12:27:39 oak-gw06 kernel: LustreError: 14269:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 12:27:39 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 12:27:39 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 12:32:46 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 12:32:46 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 12:32:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502393266, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88013da50000/0xf077f1a82e0e2649 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb98bdd14 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 12:32:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 12:32:46 oak-gw06 kernel: LustreError: 14288:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801c5a38600) refcount = 2 Aug 10 12:32:46 oak-gw06 kernel: LustreError: 14288:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 12:37:52 oak-gw06 kernel: LustreError: 14291:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88023b9b7480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 12:37:52 oak-gw06 kernel: LustreError: 14291:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 12:37:52 oak-gw06 kernel: LustreError: 14291:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88023b9b7480) refcount = 2 Aug 10 12:37:52 oak-gw06 kernel: LustreError: 14291:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 12:37:52 oak-gw06 kernel: LustreError: 14291:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88013da50800/0xf077f1a82e0e3b1f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb98c5166 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 12:37:52 oak-gw06 kernel: LustreError: 14291:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 12:37:52 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 12:37:52 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 12:43:02 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 12:43:02 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 12:43:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502393882, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88011abe9400/0xf077f1a82e0e5494 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb98ccea9 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 12:43:02 oak-gw06 kernel: LustreError: 14311:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88042b53e3c0) refcount = 2 Aug 10 12:43:02 oak-gw06 kernel: LustreError: 14311:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 12:43:02 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 12:48:12 oak-gw06 kernel: LustreError: 14316:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801cb7c2240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 12:48:12 oak-gw06 kernel: LustreError: 14316:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 12:48:12 oak-gw06 kernel: LustreError: 14316:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801cb7c2240) refcount = 2 Aug 10 12:48:12 oak-gw06 kernel: LustreError: 14316:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 12:48:12 oak-gw06 kernel: LustreError: 14316:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88038b726200/0xf077f1a82e0e811f lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb98d45be expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 12:48:12 oak-gw06 kernel: LustreError: 14316:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 12:48:12 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 12:48:12 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 12:53:23 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 12:53:23 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 12:53:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502394503, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88038b727a00/0xf077f1a82e0e9195 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb98dc2ad expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 12:53:23 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 12:53:23 oak-gw06 kernel: LustreError: 14328:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880374aea3c0) refcount = 2 Aug 10 12:53:23 oak-gw06 kernel: LustreError: 14328:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 12:58:35 oak-gw06 kernel: LustreError: 14335:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801f335d0c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 12:58:35 oak-gw06 kernel: LustreError: 14335:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 12:58:35 oak-gw06 kernel: LustreError: 14335:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801f335d0c0) refcount = 2 Aug 10 12:58:35 oak-gw06 kernel: LustreError: 14335:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 12:58:35 oak-gw06 kernel: LustreError: 14335:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880425b25400/0xf077f1a82e0ea481 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb98e39de expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 12:58:35 oak-gw06 kernel: LustreError: 14335:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 12:58:35 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 12:58:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 13:03:45 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 13:03:45 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 13:03:45 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502395125, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88008353a800/0xf077f1a82e0ed8f3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb98eb88d expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 13:03:45 oak-gw06 kernel: LustreError: 14382:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880102ae30c0) refcount = 2 Aug 10 13:03:45 oak-gw06 kernel: LustreError: 14382:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 13:03:46 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 13:08:57 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 13:08:57 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 13:14:05 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 13:14:05 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 13:14:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502395745, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880242a68e00/0xf077f1a82e0f4312 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb98facf3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 13:14:05 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 13:14:05 oak-gw06 kernel: LustreError: 14417:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88012e7d6d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 13:14:05 oak-gw06 kernel: LustreError: 14417:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 13:14:05 oak-gw06 kernel: LustreError: 14417:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88012e7d6d80) refcount = 2 Aug 10 13:14:05 oak-gw06 kernel: LustreError: 14417:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 13:14:05 oak-gw06 kernel: LustreError: 14417:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880242a68e00/0xf077f1a82e0f4312 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb98facf3 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 13:14:05 oak-gw06 kernel: LustreError: 14417:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 13:19:12 oak-gw06 kernel: LustreError: 14429:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88018aae4600) refcount = 2 Aug 10 13:19:12 oak-gw06 kernel: LustreError: 14429:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 13:19:12 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 13:19:12 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 13:24:19 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 13:24:19 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 13:24:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502396359, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880223b6e000/0xf077f1a82e11281f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9909f45 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 13:24:19 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 13:24:19 oak-gw06 kernel: LustreError: 14440:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880413f39d80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 13:24:19 oak-gw06 kernel: LustreError: 14440:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 13:24:19 oak-gw06 kernel: LustreError: 14440:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880413f39d80) refcount = 2 Aug 10 13:24:19 oak-gw06 kernel: LustreError: 14440:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 13:24:19 oak-gw06 kernel: LustreError: 14440:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880223b6e000/0xf077f1a82e11281f lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9909f45 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 13:24:19 oak-gw06 kernel: LustreError: 14440:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 13:29:27 oak-gw06 kernel: LustreError: 14448:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880419ef9f00) refcount = 2 Aug 10 13:29:27 oak-gw06 kernel: LustreError: 14448:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 13:29:27 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 13:29:27 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 13:34:39 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 13:34:39 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 13:34:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502396979, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801f2671c00/0xf077f1a82e117af2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb99193d5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 13:34:39 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 13:34:39 oak-gw06 kernel: LustreError: 14476:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802b0a07300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 13:34:39 oak-gw06 kernel: LustreError: 14476:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 13:34:39 oak-gw06 kernel: LustreError: 14476:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802b0a07300) refcount = 2 Aug 10 13:34:39 oak-gw06 kernel: LustreError: 14476:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 13:34:39 oak-gw06 kernel: LustreError: 14476:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801f2671c00/0xf077f1a82e117af2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb99193d5 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 13:34:39 oak-gw06 kernel: LustreError: 14476:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 13:39:54 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 13:39:54 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 13:45:03 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 13:45:03 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 13:45:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502397603, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88010cca3800/0xf077f1a82e13ad53 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9928a80 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 13:45:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 13:45:03 oak-gw06 kernel: LustreError: 14517:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88012b30f480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 13:45:03 oak-gw06 kernel: LustreError: 14517:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88012b30f480) refcount = 2 Aug 10 13:45:03 oak-gw06 kernel: LustreError: 14517:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 13:45:03 oak-gw06 kernel: LustreError: 14517:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88010cca3800/0xf077f1a82e13ad53 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9928a80 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 13:50:09 oak-gw06 kernel: LustreError: 14531:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880365b8da80) refcount = 2 Aug 10 13:50:09 oak-gw06 kernel: LustreError: 14531:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 13:50:09 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 13:50:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 13:55:21 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 13:55:21 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 13:55:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502398221, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880105940000/0xf077f1a82e1599a6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9937c8c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 13:55:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 13:55:21 oak-gw06 kernel: LustreError: 14535:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88012e4f7300) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 13:55:21 oak-gw06 kernel: LustreError: 14535:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 13:55:21 oak-gw06 kernel: LustreError: 14535:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88012e4f7300) refcount = 2 Aug 10 13:55:21 oak-gw06 kernel: LustreError: 14535:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 13:55:21 oak-gw06 kernel: LustreError: 14535:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880105940000/0xf077f1a82e1599a6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9937c8c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 13:55:21 oak-gw06 kernel: LustreError: 14535:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 14:00:31 oak-gw06 kernel: LustreError: 14546:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88003e52c300) refcount = 2 Aug 10 14:00:31 oak-gw06 kernel: LustreError: 14546:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 14:00:31 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 14:00:31 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 14:05:40 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 14:05:40 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 14:05:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502398840, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88013bfc7800/0xf077f1a82e15ccf9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9947185 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 14:05:40 oak-gw06 kernel: LustreError: 14588:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801e24750c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 14:05:40 oak-gw06 kernel: LustreError: 14588:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 14:05:40 oak-gw06 kernel: LustreError: 14588:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801e24750c0) refcount = 2 Aug 10 14:05:40 oak-gw06 kernel: LustreError: 14588:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 14:05:40 oak-gw06 kernel: LustreError: 14588:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88013bfc7800/0xf077f1a82e15ccf9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9947185 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 14:05:40 oak-gw06 kernel: LustreError: 14588:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 14:05:40 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 14:10:48 oak-gw06 kernel: LustreError: 14603:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8804186cb180) refcount = 2 Aug 10 14:10:48 oak-gw06 kernel: LustreError: 14603:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 14:10:48 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 14:10:48 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 14:15:55 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 14:15:55 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 14:15:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502399455, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880345a17200/0xf077f1a82e162eeb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9956551 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 14:15:55 oak-gw06 kernel: LustreError: 14607:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88028ff3da80) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 14:15:55 oak-gw06 kernel: LustreError: 14607:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 14:15:55 oak-gw06 kernel: LustreError: 14607:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88028ff3da80) refcount = 2 Aug 10 14:15:55 oak-gw06 kernel: LustreError: 14607:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 14:15:55 oak-gw06 kernel: LustreError: 14607:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880345a17200/0xf077f1a82e162eeb lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9956551 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 14:15:55 oak-gw06 kernel: LustreError: 14607:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 14:15:55 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 14:21:04 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 14:21:04 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 14:26:10 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 14:26:10 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 14:26:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502400070, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802627b3c00/0xf077f1a82e168808 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9965875 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 14:26:10 oak-gw06 kernel: LustreError: 14630:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88015f157480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 14:26:10 oak-gw06 kernel: LustreError: 14630:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88015f157480) refcount = 2 Aug 10 14:26:10 oak-gw06 kernel: LustreError: 14630:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 14:26:10 oak-gw06 kernel: LustreError: 14630:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8802627b3c00/0xf077f1a82e168808 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9965875 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 14:26:10 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 14:31:18 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 14:31:18 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 14:36:24 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 14:36:24 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 14:36:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502400684, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88005a100e00/0xf077f1a82e187b1c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9974aab expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 14:36:24 oak-gw06 kernel: LustreError: 14672:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880150e8f780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 14:36:24 oak-gw06 kernel: LustreError: 14672:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880150e8f780) refcount = 2 Aug 10 14:36:24 oak-gw06 kernel: LustreError: 14672:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 14:36:24 oak-gw06 kernel: LustreError: 14672:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88005a100e00/0xf077f1a82e187b1c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9974aab expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 14:36:24 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 14:41:34 oak-gw06 kernel: LustreError: 14712:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880150e8f780) refcount = 2 Aug 10 14:41:34 oak-gw06 kernel: LustreError: 14712:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 14:41:34 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 14:41:35 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 14:46:42 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 14:46:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 14:46:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502401302, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803c3106e00/0xf077f1a82e1b642c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9983e3f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 14:46:42 oak-gw06 kernel: LustreError: 14721:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88028a7c7240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 14:46:42 oak-gw06 kernel: LustreError: 14721:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 14:46:42 oak-gw06 kernel: LustreError: 14721:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88028a7c7240) refcount = 2 Aug 10 14:46:42 oak-gw06 kernel: LustreError: 14721:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 14:46:42 oak-gw06 kernel: LustreError: 14721:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803c3106e00/0xf077f1a82e1b642c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9983e3f expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 14:46:42 oak-gw06 kernel: LustreError: 14721:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 14:46:42 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 14:51:51 oak-gw06 kernel: LustreError: 14740:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801d36ab300) refcount = 2 Aug 10 14:51:51 oak-gw06 kernel: LustreError: 14740:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 14:51:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 14:51:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 14:57:01 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 14:57:01 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 14:57:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502401921, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880232d55a00/0xf077f1a82e1c79af lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb999338c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 14:57:01 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 14:57:01 oak-gw06 kernel: LustreError: 14752:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801e2ebf480) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 14:57:01 oak-gw06 kernel: LustreError: 14752:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 14:57:01 oak-gw06 kernel: LustreError: 14752:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801e2ebf480) refcount = 2 Aug 10 14:57:01 oak-gw06 kernel: LustreError: 14752:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 14:57:01 oak-gw06 kernel: LustreError: 14752:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880232d55a00/0xf077f1a82e1c79af lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb999338c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 14:57:01 oak-gw06 kernel: LustreError: 14752:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 15:02:09 oak-gw06 kernel: LustreError: 14799:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88040b37c180) refcount = 2 Aug 10 15:02:09 oak-gw06 kernel: LustreError: 14799:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 15:02:09 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 15:02:09 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 15:07:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 15:07:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 15:07:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502402536, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880399cd9a00/0xf077f1a82e1d6302 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb99a26e1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 15:07:16 oak-gw06 kernel: LustreError: 14806:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880071467840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 15:07:16 oak-gw06 kernel: LustreError: 14806:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 15:07:16 oak-gw06 kernel: LustreError: 14806:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880071467840) refcount = 2 Aug 10 15:07:16 oak-gw06 kernel: LustreError: 14806:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 15:07:16 oak-gw06 kernel: LustreError: 14806:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880399cd9a00/0xf077f1a82e1d6302 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb99a26e1 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 15:07:16 oak-gw06 kernel: LustreError: 14806:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 15:07:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 15:12:28 oak-gw06 kernel: LustreError: 14821:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801250886c0) refcount = 2 Aug 10 15:12:28 oak-gw06 kernel: LustreError: 14821:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 15:12:28 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 15:12:28 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 15:17:38 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 15:17:38 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 15:17:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502403158, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8801e63bca00/0xf077f1a82e1df07c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb99b1c66 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 15:17:38 oak-gw06 kernel: LustreError: 14828:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88003e52c780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 15:17:38 oak-gw06 kernel: LustreError: 14828:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 15:17:38 oak-gw06 kernel: LustreError: 14828:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88003e52c780) refcount = 2 Aug 10 15:17:38 oak-gw06 kernel: LustreError: 14828:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 15:17:38 oak-gw06 kernel: LustreError: 14828:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8801e63bca00/0xf077f1a82e1df07c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb99b1c66 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 15:17:38 oak-gw06 kernel: LustreError: 14828:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 15:17:38 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 15:22:51 oak-gw06 kernel: LustreError: 14843:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801089a5540) refcount = 2 Aug 10 15:22:51 oak-gw06 kernel: LustreError: 14843:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 15:22:51 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 15:22:51 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 15:28:00 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 15:28:00 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 15:28:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502403780, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88004d5a1e00/0xf077f1a82e1e8f1b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb99c11dd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 15:28:00 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 15:28:00 oak-gw06 kernel: LustreError: 14852:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88024db59780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 15:28:00 oak-gw06 kernel: LustreError: 14852:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 15:28:00 oak-gw06 kernel: LustreError: 14852:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88024db59780) refcount = 2 Aug 10 15:28:00 oak-gw06 kernel: LustreError: 14852:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 15:28:00 oak-gw06 kernel: LustreError: 14852:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88004d5a1e00/0xf077f1a82e1e8f1b lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb99c11dd expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 15:28:00 oak-gw06 kernel: LustreError: 14852:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 15:33:10 oak-gw06 kernel: LustreError: 14863:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880167dcb240) refcount = 2 Aug 10 15:33:10 oak-gw06 kernel: LustreError: 14863:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 15:33:10 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 15:33:10 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 15:38:16 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 15:38:16 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 15:38:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502404396, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880140bd2c00/0xf077f1a82e1f1931 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb99d041a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 15:38:16 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 15:38:16 oak-gw06 kernel: LustreError: 14880:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff880167dcb780) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 15:38:16 oak-gw06 kernel: LustreError: 14880:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 15:38:16 oak-gw06 kernel: LustreError: 14880:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880167dcb780) refcount = 2 Aug 10 15:38:16 oak-gw06 kernel: LustreError: 14880:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 15:38:16 oak-gw06 kernel: LustreError: 14880:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880140bd2c00/0xf077f1a82e1f1931 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb99d041a expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 15:38:16 oak-gw06 kernel: LustreError: 14880:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 15:43:24 oak-gw06 kernel: LustreError: 14910:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803097bb540) refcount = 2 Aug 10 15:43:24 oak-gw06 kernel: LustreError: 14910:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 15:43:24 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 15:43:24 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 15:48:31 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 15:48:31 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 15:48:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502405011, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8803a86d7e00/0xf077f1a82e21a45b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb99df714 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 15:48:31 oak-gw06 kernel: LustreError: 14914:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88025333d240) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 15:48:31 oak-gw06 kernel: LustreError: 14914:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 15:48:31 oak-gw06 kernel: LustreError: 14914:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88025333d240) refcount = 2 Aug 10 15:48:31 oak-gw06 kernel: LustreError: 14914:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 15:48:31 oak-gw06 kernel: LustreError: 14914:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803a86d7e00/0xf077f1a82e21a45b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb99df714 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 15:48:31 oak-gw06 kernel: LustreError: 14914:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 15:48:31 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 15:53:37 oak-gw06 kernel: LustreError: 14930:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802adf106c0) refcount = 2 Aug 10 15:53:37 oak-gw06 kernel: LustreError: 14930:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 15:53:37 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 15:53:37 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 15:58:48 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 15:58:48 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 15:58:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502405628, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880140aa4000/0xf077f1a82e21e171 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb99ee7de expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 15:58:48 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 16:03:53 oak-gw06 kernel: LustreError: 14997:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802a5a520c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 16:03:53 oak-gw06 kernel: LustreError: 14997:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 16:03:53 oak-gw06 kernel: LustreError: 14997:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802a5a520c0) refcount = 2 Aug 10 16:03:53 oak-gw06 kernel: LustreError: 14997:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 16:03:53 oak-gw06 kernel: LustreError: 14997:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8800438e5e00/0xf077f1a82e21f04a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb99f5f5c expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 16:03:53 oak-gw06 kernel: LustreError: 14997:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 16:03:53 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 16:03:53 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 16:09:04 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 16:09:04 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 16:09:04 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502406244, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88019cd45800/0xf077f1a82e220d5b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb99fd7eb expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 16:09:04 oak-gw06 kernel: LustreError: 15005:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803cefc0780) refcount = 2 Aug 10 16:09:04 oak-gw06 kernel: LustreError: 15005:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 16:09:04 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 16:14:14 oak-gw06 kernel: LustreError: 15018:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8802d9d3c900) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 16:14:14 oak-gw06 kernel: LustreError: 15018:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 16:14:14 oak-gw06 kernel: LustreError: 15018:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8802d9d3c900) refcount = 2 Aug 10 16:14:14 oak-gw06 kernel: LustreError: 15018:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 16:14:14 oak-gw06 kernel: LustreError: 15018:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88019cd44e00/0xf077f1a82e2220e1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9a04fd2 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 16:14:14 oak-gw06 kernel: LustreError: 15018:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 16:14:14 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 16:14:14 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 16:19:25 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 16:19:25 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 16:19:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502406865, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880307ad5600/0xf077f1a82e223f6c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9a0cc97 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 16:19:25 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 16:19:25 oak-gw06 kernel: LustreError: 15036:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff880122697b40) refcount = 2 Aug 10 16:19:25 oak-gw06 kernel: LustreError: 15036:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 16:24:36 oak-gw06 kernel: LustreError: 15052:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88026bac9540) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 16:24:36 oak-gw06 kernel: LustreError: 15052:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 16:24:36 oak-gw06 kernel: LustreError: 15052:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88026bac9540) refcount = 2 Aug 10 16:24:36 oak-gw06 kernel: LustreError: 15052:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 16:24:36 oak-gw06 kernel: LustreError: 15052:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880038386e00/0xf077f1a82e227ad0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9a143e4 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 16:24:36 oak-gw06 kernel: LustreError: 15052:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 16:24:36 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 16:24:36 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 16:29:43 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 16:29:43 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 16:29:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502407483, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff880002e0fe00/0xf077f1a82e22f5f8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9a1c120 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 16:29:43 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 16:29:43 oak-gw06 kernel: LustreError: 15066:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8803aeb67780) refcount = 2 Aug 10 16:29:43 oak-gw06 kernel: LustreError: 15066:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 16:34:53 oak-gw06 kernel: LustreError: 15232:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88012a6bf0c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 16:34:53 oak-gw06 kernel: LustreError: 15232:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 16:34:53 oak-gw06 kernel: LustreError: 15232:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88012a6bf0c0) refcount = 2 Aug 10 16:34:53 oak-gw06 kernel: LustreError: 15232:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 16:34:53 oak-gw06 kernel: LustreError: 15232:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8803ac58ba00/0xf077f1a82e23b2e3 lrc: 2/0,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9a23835 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 16:34:53 oak-gw06 kernel: LustreError: 15232:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 16:34:53 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 16:34:53 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 16:40:03 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 16:40:03 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 16:40:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502408103, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8802390fc800/0xf077f1a82e246015 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9a2b491 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 16:40:03 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 16:40:03 oak-gw06 kernel: LustreError: 15255:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88018235ff00) refcount = 2 Aug 10 16:40:03 oak-gw06 kernel: LustreError: 15255:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 16:45:11 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 16:45:11 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 16:50:21 oak-gw06 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Aug 10 16:50:21 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 10 16:50:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1502408721, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff88014d6cee00/0xf077f1a82e260d6f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x7f86053cb9a3a7d8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 16:50:21 oak-gw06 kernel: LustreError: 15293:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8801b580e000) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 16:50:21 oak-gw06 kernel: LustreError: 15293:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 10 16:50:21 oak-gw06 kernel: LustreError: 15293:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8801b580e000) refcount = 2 Aug 10 16:50:21 oak-gw06 kernel: LustreError: 15293:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 16:50:21 oak-gw06 kernel: LustreError: 15293:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff88014d6cee00/0xf077f1a82e260d6f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9a3a7d8 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 16:50:21 oak-gw06 kernel: LustreError: 15293:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 10 16:50:21 oak-gw06 kernel: LustreError: 1790:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message Aug 10 16:55:37 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502409330/real 1502409330] req@ffff8802e82a8900 x1566269322149472/t0(0) o250->MGC10.0.2.51@o2ib5@10.0.2.51@o2ib5:26/25 lens 520/544 e 0 to 1 dl 1502409337 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 10 16:55:37 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Aug 10 16:56:02 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502409355/real 1502409355] req@ffff8802526aed00 x1566269322224848/t0(0) o250->MGC10.0.2.51@o2ib5@10.0.2.52@o2ib5:26/25 lens 520/544 e 0 to 1 dl 1502409362 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 10 16:56:32 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502409380/real 1502409380] req@ffff8802b770de00 x1566269322310496/t0(0) o250->MGC10.0.2.51@o2ib5@10.0.2.51@o2ib5:26/25 lens 520/544 e 0 to 1 dl 1502409392 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 10 16:57:27 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502409430/real 1502409430] req@ffff8801a26ebc00 x1566269323057296/t0(0) o250->MGC10.0.2.51@o2ib5@10.0.2.51@o2ib5:26/25 lens 520/544 e 0 to 1 dl 1502409447 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 10 16:57:27 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 10 16:58:47 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502409505/real 1502409505] req@ffff8801f064e400 x1566269324977968/t0(0) o250->MGC10.0.2.51@o2ib5@10.0.2.52@o2ib5:26/25 lens 520/544 e 0 to 1 dl 1502409527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 10 16:58:47 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Aug 10 17:01:52 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502409680/real 1502409680] req@ffff88010ab6b300 x1566269326017584/t0(0) o250->MGC10.0.2.51@o2ib5@10.0.2.52@o2ib5:26/25 lens 520/544 e 0 to 1 dl 1502409712 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 10 17:01:52 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Aug 10 17:07:07 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502409980/real 1502409980] req@ffff88013fbe7c00 x1566269330247680/t0(0) o250->MGC10.0.2.51@o2ib5@10.0.2.52@o2ib5:26/25 lens 520/544 e 0 to 1 dl 1502410027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 10 17:07:07 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Aug 10 17:11:20 oak-gw06 kernel: Lustre: Evicted from MGS (at 10.0.2.51@o2ib5) after server handle changed from 0x7f86053cb7f7e12a to 0x19ecaeeddb5de38d Aug 10 17:11:20 oak-gw06 kernel: LustreError: 15681:0:(ldlm_resource.c:882:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff88010470dcc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 10 17:11:20 oak-gw06 kernel: LustreError: 15681:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff88010470dcc0) refcount = 2 Aug 10 17:11:20 oak-gw06 kernel: LustreError: 15681:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 10 17:11:20 oak-gw06 kernel: LustreError: 15681:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff880008037c00/0xf077f1a82e27bc82 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x7f86053cb9a41f48 expref: -99 pid: 1790 timeout: 0 lvb_type: 0 Aug 10 17:11:20 oak-gw06 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) Aug 10 17:11:20 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 10 21:52:09 oak-gw06 kernel: Lustre: DEBUG MARKER: Thu Aug 10 21:52:09 2017 Aug 12 06:08:39 oak-gw06 kernel: Lustre: 23358:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502543312/real 1502543312] req@ffff880117bd3900 x1566269766329552/t0(0) o101->oak-OST0018-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 328/400 e 0 to 1 dl 1502543319 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 12 06:08:39 oak-gw06 kernel: Lustre: 23358:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Aug 12 06:08:39 oak-gw06 kernel: Lustre: oak-OST0018-osc-ffff88041b99c000: Connection to oak-OST0018 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 12 06:08:39 oak-gw06 kernel: Lustre: Skipped 15 previous similar messages Aug 12 06:08:41 oak-gw06 kernel: Lustre: oak-OST0028-osc-ffff88041b99c000: Connection to oak-OST0028 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 12 06:08:44 oak-gw06 kernel: Lustre: oak-OST0012-osc-ffff88041b99c000: Connection to oak-OST0012 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 12 06:08:44 oak-gw06 kernel: Lustre: Skipped 2 previous similar messages Aug 12 06:09:04 oak-gw06 kernel: Lustre: oak-OST0008-osc-ffff88041b99c000: Connection to oak-OST0008 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 12 06:09:04 oak-gw06 kernel: Lustre: Skipped 9 previous similar messages Aug 12 06:09:35 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3126:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 11 seconds Aug 12 06:09:35 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3189:kiblnd_check_conns()) Timed out RDMA with 10.0.2.101@o2ib5 (61): c: 0, oc: 0, rc: 8 Aug 12 06:13:22 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1502543602/real 1502543602] req@ffff8800afd26700 x1566269766349088/t0(0) o8->oak-OST0002-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1502543613 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 12 06:13:22 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 98 previous similar messages Aug 12 06:16:42 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1502543802/real 1502543802] req@ffff8801753c5200 x1566269766359024/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1502543833 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 12 06:16:42 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 95 previous similar messages Aug 12 06:21:42 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1502544102/real 1502544102] req@ffff880277b9a400 x1566269766373472/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1502544157 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 12 06:21:42 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Aug 12 06:31:42 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1502544702/real 1502544702] req@ffff8801ab2ba700 x1566269766405312/t0(0) o8->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1502544718 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 12 06:31:42 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 295 previous similar messages Aug 12 06:42:32 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1502545352/real 1502545352] req@ffff880024056700 x1566269766441104/t0(0) o8->oak-OST000a-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1502545408 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 12 06:42:32 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 303 previous similar messages Aug 12 06:53:03 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502545952/real 1502545952] req@ffff88012f4d8000 x1566269766471104/t0(0) o8->oak-OST0018-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1502545983 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 12 06:53:03 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 287 previous similar messages Aug 12 06:54:22 oak-gw06 kernel: Lustre: oak-OST000e-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 12 06:55:32 oak-gw06 kernel: Lustre: oak-OST0022-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 12 06:55:32 oak-gw06 kernel: Lustre: Skipped 22 previous similar messages Aug 14 20:15:03 oak-gw06 kernel: ptlrpcd_00_01: page allocation failure: order:2, mode:0x104020 Aug 14 20:15:03 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 14 20:15:03 oak-gw06 kernel: CPU: 4 PID: 3816 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:15:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:15:03 oak-gw06 kernel: 0000000000104020 00000000d75a30cf ffff88043fd039d8 ffffffff8168662f Aug 14 20:15:03 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 14 20:15:03 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff880418bf6c80 00000000d75a30cf Aug 14 20:15:03 oak-gw06 kernel: Call Trace: Aug 14 20:15:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:15:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:15:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:15:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:15:03 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:15:03 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:15:03 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:15:03 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:15:03 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:15:03 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:15:03 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:15:03 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:15:03 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:15:03 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:15:03 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:15:03 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:15:03 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:15:03 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:15:03 oak-gw06 kernel: [] ? cl_page_assume+0x77/0x200 [obdclass] Aug 14 20:15:03 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 14 20:15:03 oak-gw06 kernel: [] ? iov_iter_copy_from_user_atomic+0x6a/0xa0 Aug 14 20:15:03 oak-gw06 kernel: [] generic_file_buffered_write+0x155/0x2a0 Aug 14 20:15:03 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 14 20:15:03 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 14 20:15:03 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 14 20:15:03 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 14 20:15:03 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 14 20:15:03 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 14 20:15:03 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 14 20:15:03 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 14 20:15:03 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 14 20:15:03 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 14 20:15:03 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 14 20:15:03 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 14 20:15:03 oak-gw06 kernel: CPU: 4 PID: 3816 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:15:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:15:03 oak-gw06 kernel: 0000000000104020 00000000d75a30cf ffff88043fd039d8 ffffffff8168662f Aug 14 20:15:03 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 14 20:15:03 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff880418bf6c80 00000000d75a30cf Aug 14 20:15:03 oak-gw06 kernel: Call Trace: Aug 14 20:15:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:15:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:15:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:15:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:15:03 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:15:03 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:15:03 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:15:03 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:15:03 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:15:03 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:15:03 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:15:03 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:15:03 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:15:03 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:15:03 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:15:03 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:15:03 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:15:03 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:15:03 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:15:03 oak-gw06 kernel: [] ? cl_page_assume+0x77/0x200 [obdclass] Aug 14 20:15:03 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 14 20:15:03 oak-gw06 kernel: [] ? iov_iter_copy_from_user_atomic+0x6a/0xa0 Aug 14 20:15:03 oak-gw06 kernel: [] generic_file_buffered_write+0x155/0x2a0 Aug 14 20:15:03 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 14 20:15:03 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 14 20:15:03 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 14 20:15:03 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 14 20:15:03 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 14 20:15:03 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 14 20:15:03 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 14 20:15:03 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 14 20:15:03 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 14 20:15:03 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 14 20:15:03 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 14 20:15:03 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 14 20:15:03 oak-gw06 kernel: CPU: 4 PID: 3816 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:15:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:15:03 oak-gw06 kernel: 0000000000104020 00000000d75a30cf ffff88043fd039d8 ffffffff8168662f Aug 14 20:15:03 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff88043fd03a80 ffffffff815d720c Aug 14 20:15:03 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffffffff81aebbd0 00000000d75a30cf Aug 14 20:15:03 oak-gw06 kernel: Call Trace: Aug 14 20:15:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:15:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:15:03 oak-gw06 kernel: [] ? tcp_v4_rcv+0x7ac/0x9a0 Aug 14 20:15:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:15:03 oak-gw06 kernel: [] ? ip_local_deliver_finish+0xb4/0x1f0 Aug 14 20:15:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:15:03 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:15:03 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:15:03 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:15:03 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:15:03 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:15:03 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:15:03 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:15:03 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:15:03 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:15:03 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:15:03 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:15:03 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:15:03 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:15:03 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:15:03 oak-gw06 kernel: [] ? cl_page_assume+0x77/0x200 [obdclass] Aug 14 20:15:03 oak-gw06 kernel: [] ? copy_user_enhanced_fast_string+0x9/0x20 Aug 14 20:15:03 oak-gw06 kernel: [] ? iov_iter_copy_from_user_atomic+0x6a/0xa0 Aug 14 20:15:03 oak-gw06 kernel: [] generic_file_buffered_write+0x155/0x2a0 Aug 14 20:15:03 oak-gw06 kernel: [] __generic_file_aio_write+0x1e2/0x400 Aug 14 20:15:03 oak-gw06 kernel: [] ? lov_lsm_addref+0x86/0x210 [lov] Aug 14 20:15:03 oak-gw06 kernel: [] vvp_io_write_start+0x2bb/0x720 [lustre] Aug 14 20:15:03 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 14 20:15:03 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 14 20:15:03 oak-gw06 kernel: [] ll_file_io_generic+0x67f/0xb50 [lustre] Aug 14 20:15:03 oak-gw06 kernel: [] ll_file_aio_write+0x12d/0x1f0 [lustre] Aug 14 20:15:03 oak-gw06 kernel: [] ll_file_write+0xce/0x1e0 [lustre] Aug 14 20:15:03 oak-gw06 kernel: [] vfs_write+0xbd/0x1e0 Aug 14 20:15:03 oak-gw06 kernel: [] SyS_write+0x7f/0xe0 Aug 14 20:15:03 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 14 20:15:03 oak-gw06 kernel: CPU: 3 PID: 1763 Comm: ptlrpcd_00_01 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:15:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:15:03 oak-gw06 kernel: 0000000000104020 00000000e73c1259 ffff88043fcc39d8 ffffffff8168662f Aug 14 20:15:03 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000046082a7fd630 0000000000000001 Aug 14 20:15:03 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff880023409f00 00000000e73c1259 Aug 14 20:15:03 oak-gw06 kernel: Call Trace: Aug 14 20:15:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:15:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:15:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:15:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:15:03 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:15:03 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:15:03 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:15:03 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:15:03 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:15:03 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:15:03 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:15:03 oak-gw06 kernel: [] ? __dev_kfree_skb_any+0x3d/0x50 Aug 14 20:15:03 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:15:03 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:15:03 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:15:03 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:15:03 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:15:03 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:15:03 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:15:03 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:15:03 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 14 20:15:03 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 14 20:15:03 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 14 20:15:03 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 14 20:15:03 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 14 20:15:03 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 14 20:15:03 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 14 20:15:03 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 14 20:15:03 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 14 20:15:03 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 14 20:15:03 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 14 20:15:03 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 14 20:15:03 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 14 20:15:03 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:15:03 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:15:03 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:15:03 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 14 20:15:03 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:15:03 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:15:03 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:15:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:15:03 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:15:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:12 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:12 oak-gw06 kernel: CPU: 4 PID: 3872 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:12 oak-gw06 kernel: 0000000000104020 00000000409aab66 ffff88043fd039d8 ffffffff8168662f Aug 14 20:22:12 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 14 20:22:12 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff880418bf45c0 00000000409aab66 Aug 14 20:22:12 oak-gw06 kernel: Call Trace: Aug 14 20:22:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:12 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:12 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:12 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:12 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:12 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:12 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:12 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:12 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:12 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:12 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:12 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:12 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:12 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:22:12 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:22:12 oak-gw06 kernel: [] ? file_read_actor+0xd7/0x180 Aug 14 20:22:12 oak-gw06 kernel: [] generic_file_aio_read+0x48b/0x790 Aug 14 20:22:12 oak-gw06 kernel: [] vvp_io_read_start+0x4b4/0x5a0 [lustre] Aug 14 20:22:12 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 14 20:22:12 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 14 20:22:12 oak-gw06 kernel: [] ll_file_io_generic+0x570/0xb50 [lustre] Aug 14 20:22:12 oak-gw06 kernel: [] ll_file_aio_read+0x347/0x3e0 [lustre] Aug 14 20:22:12 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 14 20:22:12 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 14 20:22:12 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 14 20:22:12 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 14 20:22:13 oak-gw06 kernel: ptlrpcd_00_02: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:13 oak-gw06 kernel: swapper/4: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:13 oak-gw06 kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:13 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:13 oak-gw06 kernel: 0000000000104020 803b52a30d0a83c1 ffff88043fd039d8 ffffffff8168662f Aug 14 20:22:13 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffffffff815ce55f 0077d0007fc03700 Aug 14 20:22:13 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000000 803b52a30d0a83c1 Aug 14 20:22:13 oak-gw06 kernel: Call Trace: Aug 14 20:22:13 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:13 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:13 oak-gw06 kernel: [] ? tcp_transmit_skb+0x4af/0x990 Aug 14 20:22:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:13 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:13 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:13 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:13 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:13 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:13 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:13 oak-gw06 kernel: [] ? kfree_skbmem+0x37/0x90 Aug 14 20:22:13 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:22:13 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:13 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:13 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:13 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:13 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:13 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:13 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:22:13 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:22:13 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 14 20:22:13 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 14 20:22:13 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 14 20:22:13 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 14 20:22:13 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 14 20:22:13 oak-gw06 kernel: swapper/4: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:13 oak-gw06 kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:13 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:13 oak-gw06 kernel: 0000000000104020 803b52a30d0a83c1 ffff88043fd039d8 ffffffff8168662f Aug 14 20:22:13 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffffffff815ce55f 0077d0007fc03700 Aug 14 20:22:13 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000000 803b52a30d0a83c1 Aug 14 20:22:13 oak-gw06 kernel: Call Trace: Aug 14 20:22:13 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:13 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:13 oak-gw06 kernel: [] ? tcp_transmit_skb+0x4af/0x990 Aug 14 20:22:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:13 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:13 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:13 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:13 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:13 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:13 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:13 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:13 oak-gw06 kernel: [] ? kfree_skbmem+0x37/0x90 Aug 14 20:22:13 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:22:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:13 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:13 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:13 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:13 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:13 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:13 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:13 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:13 oak-gw06 kernel: [] ? kfree_skbmem+0x37/0x90 Aug 14 20:22:13 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:22:13 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:13 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:13 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:13 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:13 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:13 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:13 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:22:13 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:22:13 oak-gw06 kernel: [] ? native_safe_halt+0x6/0x10 Aug 14 20:22:13 oak-gw06 kernel: [] default_idle+0x1f/0xc0 Aug 14 20:22:13 oak-gw06 kernel: [] arch_cpu_idle+0x26/0x30 Aug 14 20:22:13 oak-gw06 kernel: [] cpu_startup_entry+0x245/0x290 Aug 14 20:22:13 oak-gw06 kernel: [] start_secondary+0x1ba/0x230 Aug 14 20:22:13 oak-gw06 kernel: swapper/4: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:13 oak-gw06 kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:13 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:13 oak-gw06 kernel: 0000000000104020 803b52a30d0a83c1 ffff88043fd039d8 ffffffff8168662f Aug 14 20:22:13 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffffffff815ce55f 0077d0007fc03700 Aug 14 20:22:13 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000000 803b52a30d0a83c1 Aug 14 20:22:13 oak-gw06 kernel: Call Trace: Aug 14 20:22:13 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:13 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:13 oak-gw06 kernel: [] ? tcp_transmit_skb+0x4af/0x990 Aug 14 20:22:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:13 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:13 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:13 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:13 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:13 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:13 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:13 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:13 oak-gw06 kernel: [] ? kfree_skbmem+0x37/0x90 Aug 14 20:22:13 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:22:13 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:13 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:13 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:13 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:13 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:13 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:13 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:22:13 oak-gw06 kernel: CPU: 3 PID: 1764 Comm: ptlrpcd_00_02 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:13 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:13 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88002340cd80 000000000ae3cb3e Aug 14 20:22:13 oak-gw06 kernel: Call Trace: Aug 14 20:22:13 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:13 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:13 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:13 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:13 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:13 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:13 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:13 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:13 oak-gw06 kernel: [] cl_page_put+0x6a/0x3d0 [obdclass] Aug 14 20:22:13 oak-gw06 kernel: [] ? cl_page_disown+0x43/0x120 [obdclass] Aug 14 20:22:13 oak-gw06 kernel: [] discard_pagevec+0x79/0xd0 [osc] Aug 14 20:22:13 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 14 20:22:13 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 14 20:22:13 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 14 20:22:13 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:22:13 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:22:13 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:22:13 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 14 20:22:13 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:22:13 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:22:13 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:13 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:13 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: ldlm_bl_99: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:19 oak-gw06 kernel: ldlm_bl_107: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:19 oak-gw06 kernel: CPU: 3 PID: 2098 Comm: ldlm_bl_107 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:19 oak-gw06 kernel: 0000000000104020 000000006d175c87 ffff88043fcc39f8 ffffffff8168662f Aug 14 20:22:19 oak-gw06 kernel: ffff88043fcc3a88 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 14 20:22:19 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff880023409740 000000006d175c87 Aug 14 20:22:19 oak-gw06 kernel: Call Trace: Aug 14 20:22:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:19 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:19 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:19 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:22:19 oak-gw06 kernel: [] ? ttwu_do_wakeup+0x19/0xd0 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:19 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:19 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:19 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:19 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:19 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 14 20:22:19 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 14 20:22:19 oak-gw06 kernel: [] ? osc_ldlm_blocking_ast+0x59/0x3a0 [osc] Aug 14 20:22:19 oak-gw06 kernel: [] ? cl_object_get+0x5/0x50 [obdclass] Aug 14 20:22:19 oak-gw06 kernel: [] ? osc_ldlm_blocking_ast+0x27a/0x3a0 [osc] Aug 14 20:22:19 oak-gw06 kernel: [] ldlm_cancel_callback+0x8a/0x2e0 [ptlrpc] Aug 14 20:22:19 oak-gw06 kernel: [] ? class_handle_unhash+0x3a/0x40 [obdclass] Aug 14 20:22:19 oak-gw06 kernel: [] ldlm_cli_cancel_local+0xa0/0x420 [ptlrpc] Aug 14 20:22:19 oak-gw06 kernel: [] ldlm_cli_cancel_list_local+0xea/0x280 [ptlrpc] Aug 14 20:22:19 oak-gw06 kernel: [] ldlm_bl_thread_main+0x2c1/0x700 [ptlrpc] Aug 14 20:22:19 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:22:19 oak-gw06 kernel: [] ? ldlm_handle_bl_callback+0x410/0x410 [ptlrpc] Aug 14 20:22:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: ksoftirqd/3: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:19 oak-gw06 kernel: CPU: 3 PID: 23 Comm: ksoftirqd/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:19 oak-gw06 kernel: 0000000000104020 00000000a832c540 ffff88017a283938 ffffffff8168662f Aug 14 20:22:19 oak-gw06 kernel: ffff88017a2839c8 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 14 20:22:19 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88002340ec80 00000000a832c540 Aug 14 20:22:19 oak-gw06 kernel: Call Trace: Aug 14 20:22:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:19 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:19 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:19 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:19 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:19 oak-gw06 kernel: [] run_ksoftirqd+0x38/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] smpboot_thread_fn+0x12f/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] ? lg_double_unlock+0x90/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: ksoftirqd/3: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:19 oak-gw06 kernel: CPU: 3 PID: 23 Comm: ksoftirqd/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:19 oak-gw06 kernel: 0000000000104020 00000000a832c540 ffff88017a283938 ffffffff8168662f Aug 14 20:22:19 oak-gw06 kernel: ffff88017a2839c8 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 14 20:22:19 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88002340ec80 00000000a832c540 Aug 14 20:22:19 oak-gw06 kernel: Call Trace: Aug 14 20:22:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:19 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:19 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:19 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:19 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:19 oak-gw06 kernel: [] run_ksoftirqd+0x38/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] smpboot_thread_fn+0x12f/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] ? lg_double_unlock+0x90/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: ksoftirqd/3: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:19 oak-gw06 kernel: CPU: 3 PID: 23 Comm: ksoftirqd/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:19 oak-gw06 kernel: 0000000000104020 00000000a832c540 ffff88017a283938 ffffffff8168662f Aug 14 20:22:19 oak-gw06 kernel: ffff88017a2839c8 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 14 20:22:19 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88002340ec80 00000000a832c540 Aug 14 20:22:19 oak-gw06 kernel: Call Trace: Aug 14 20:22:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:19 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:19 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:19 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:19 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:19 oak-gw06 kernel: [] run_ksoftirqd+0x38/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] smpboot_thread_fn+0x12f/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] ? lg_double_unlock+0x90/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: ksoftirqd/3: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:19 oak-gw06 kernel: CPU: 3 PID: 23 Comm: ksoftirqd/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:19 oak-gw06 kernel: 0000000000104020 00000000a832c540 ffff88017a283938 ffffffff8168662f Aug 14 20:22:19 oak-gw06 kernel: ffff88017a2839c8 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 14 20:22:19 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88002340ec80 00000000a832c540 Aug 14 20:22:19 oak-gw06 kernel: Call Trace: Aug 14 20:22:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:19 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:19 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:19 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:19 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:19 oak-gw06 kernel: [] run_ksoftirqd+0x38/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] smpboot_thread_fn+0x12f/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] ? lg_double_unlock+0x90/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: ksoftirqd/3: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:19 oak-gw06 kernel: CPU: 3 PID: 23 Comm: ksoftirqd/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:19 oak-gw06 kernel: 0000000000104020 00000000a832c540 ffff88017a283938 ffffffff8168662f Aug 14 20:22:19 oak-gw06 kernel: ffff88017a2839c8 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 14 20:22:19 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88002340ec80 00000000a832c540 Aug 14 20:22:19 oak-gw06 kernel: Call Trace: Aug 14 20:22:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:19 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:19 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:19 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:19 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:19 oak-gw06 kernel: [] run_ksoftirqd+0x38/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] smpboot_thread_fn+0x12f/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] ? lg_double_unlock+0x90/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: ksoftirqd/3: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:19 oak-gw06 kernel: CPU: 3 PID: 23 Comm: ksoftirqd/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:19 oak-gw06 kernel: 0000000000104020 00000000a832c540 ffff88017a283938 ffffffff8168662f Aug 14 20:22:19 oak-gw06 kernel: ffff88017a2839c8 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 14 20:22:19 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88002340ec80 00000000a832c540 Aug 14 20:22:19 oak-gw06 kernel: Call Trace: Aug 14 20:22:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:19 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:19 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:19 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:19 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:19 oak-gw06 kernel: [] run_ksoftirqd+0x38/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] smpboot_thread_fn+0x12f/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] ? lg_double_unlock+0x90/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: ksoftirqd/3: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:19 oak-gw06 kernel: CPU: 3 PID: 23 Comm: ksoftirqd/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:19 oak-gw06 kernel: 0000000000104020 00000000a832c540 ffff88017a283938 ffffffff8168662f Aug 14 20:22:19 oak-gw06 kernel: ffff88017a2839c8 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 14 20:22:19 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88002340ec80 00000000a832c540 Aug 14 20:22:19 oak-gw06 kernel: Call Trace: Aug 14 20:22:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:19 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:19 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:19 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:19 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:19 oak-gw06 kernel: [] run_ksoftirqd+0x38/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] smpboot_thread_fn+0x12f/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] ? lg_double_unlock+0x90/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: ksoftirqd/3: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:19 oak-gw06 kernel: CPU: 3 PID: 23 Comm: ksoftirqd/3 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:19 oak-gw06 kernel: 0000000000104020 00000000a832c540 ffff88017a283938 ffffffff8168662f Aug 14 20:22:19 oak-gw06 kernel: ffff88017a2839c8 ffffffff81186ba0 ffff88042a7fd630 0000000000000001 Aug 14 20:22:19 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88002340ec80 00000000a832c540 Aug 14 20:22:19 oak-gw06 kernel: Call Trace: Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:19 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:19 oak-gw06 kernel: [] run_ksoftirqd+0x38/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] smpboot_thread_fn+0x12f/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] ? lg_double_unlock+0x90/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: CPU: 4 PID: 2090 Comm: ldlm_bl_99 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:19 oak-gw06 kernel: 0000000000104020 00000000adbaaa78 ffff88043fd039d8 ffffffff8168662f Aug 14 20:22:19 oak-gw06 kernel: ffff88043fd03a68 ffffffff81186ba0 ffff880418bf64c0 ffff880068200064 Aug 14 20:22:19 oak-gw06 kernel: fffffffffffffffc 0010402000000000 000046082a7fd630 00000000adbaaa78 Aug 14 20:22:19 oak-gw06 kernel: Call Trace: Aug 14 20:22:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:19 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:19 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:19 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:22:19 oak-gw06 kernel: [] ? __dev_kfree_skb_any+0x3d/0x50 Aug 14 20:22:19 oak-gw06 kernel: [] ? task_tick_fair+0x4bf/0x680 Aug 14 20:22:19 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:19 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:19 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:19 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:19 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:19 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:19 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:22:19 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:22:19 oak-gw06 kernel: [] ? _raw_spin_unlock_irqrestore+0x1b/0x40 Aug 14 20:22:19 oak-gw06 kernel: [] remove_wait_queue+0x31/0x40 Aug 14 20:22:19 oak-gw06 kernel: [] ldlm_bl_thread_main+0x4b4/0x700 [ptlrpc] Aug 14 20:22:19 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:22:19 oak-gw06 kernel: [] ? ldlm_handle_bl_callback+0x410/0x410 [ptlrpc] Aug 14 20:22:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: warn_alloc_failed: 927 callbacks suppressed Aug 14 20:22:47 oak-gw06 kernel: ptlrpcd_00_03: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:47 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:47 oak-gw06 kernel: CPU: 4 PID: 1765 Comm: ptlrpcd_00_03 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:47 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:47 oak-gw06 kernel: 0000000000104020 000000009d48cbc7 ffff88043fd039f8 ffffffff8168662f Aug 14 20:22:47 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff880418bf45c0 000000009d48cbc7 Aug 14 20:22:47 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000020 000000009d48cbc7 Aug 14 20:22:47 oak-gw06 kernel: Call Trace: Aug 14 20:22:47 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:47 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:47 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:47 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:47 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:47 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:47 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 14 20:22:47 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 14 20:22:47 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 14 20:22:47 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 14 20:22:47 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 14 20:22:47 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 14 20:22:47 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: ptlrpcd_00_03: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:47 oak-gw06 kernel: CPU: 4 PID: 1765 Comm: ptlrpcd_00_03 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:47 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:47 oak-gw06 kernel: 0000000000104020 000000009d48cbc7 ffff88043fd039f8 ffffffff8168662f Aug 14 20:22:47 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff880418bf45c0 000000009d48cbc7 Aug 14 20:22:47 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000020 000000009d48cbc7 Aug 14 20:22:47 oak-gw06 kernel: Call Trace: Aug 14 20:22:47 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:47 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:47 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:47 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:47 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:47 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:47 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 14 20:22:47 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 14 20:22:47 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 14 20:22:47 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 14 20:22:47 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 14 20:22:47 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 14 20:22:47 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: ptlrpcd_00_03: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:47 oak-gw06 kernel: CPU: 4 PID: 1765 Comm: ptlrpcd_00_03 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:47 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:47 oak-gw06 kernel: 0000000000104020 000000009d48cbc7 ffff88043fd039f8 ffffffff8168662f Aug 14 20:22:47 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff880418bf45c0 000000009d48cbc7 Aug 14 20:22:47 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000020 000000009d48cbc7 Aug 14 20:22:47 oak-gw06 kernel: Call Trace: Aug 14 20:22:47 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:47 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:47 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:47 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:47 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:47 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:47 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 14 20:22:47 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 14 20:22:47 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 14 20:22:47 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 14 20:22:47 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 14 20:22:47 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 14 20:22:47 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: ptlrpcd_00_03: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:47 oak-gw06 kernel: CPU: 4 PID: 1765 Comm: ptlrpcd_00_03 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:47 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:47 oak-gw06 kernel: 0000000000104020 000000009d48cbc7 ffff88043fd039f8 ffffffff8168662f Aug 14 20:22:47 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff880418bf45c0 000000009d48cbc7 Aug 14 20:22:47 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000000020 000000009d48cbc7 Aug 14 20:22:47 oak-gw06 kernel: Call Trace: Aug 14 20:22:47 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:47 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:47 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:47 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:47 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:47 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:47 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 14 20:22:47 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 14 20:22:47 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 14 20:22:47 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 14 20:22:47 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 14 20:22:47 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 14 20:22:47 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: ptlrpcd_00_03: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:47 oak-gw06 kernel: CPU: 4 PID: 1765 Comm: ptlrpcd_00_03 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:47 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:47 oak-gw06 kernel: 0000000000104020 000000009d48cbc7 ffff88043fd039b0 ffffffff8168662f Aug 14 20:22:47 oak-gw06 kernel: ffff88043fd03a40 ffffffff81186ba0 0000000000000000 ffff88043fd039e8 Aug 14 20:22:47 oak-gw06 kernel: fffffffffffffffc 0010402000000000 000000009d48cbc7 000000009d48cbc7 Aug 14 20:22:47 oak-gw06 kernel: Call Trace: Aug 14 20:22:47 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:47 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:47 oak-gw06 kernel: [] ? warn_alloc_failed+0x110/0x180 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:47 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:47 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_alloc_rx_data.isra.70+0x54/0x1c0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_rx_int+0x836/0x17b0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:47 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:47 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:47 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 14 20:22:47 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 14 20:22:47 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 14 20:22:47 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 14 20:22:47 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 14 20:22:47 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 14 20:22:47 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: ptlrpcd_00_03: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:47 oak-gw06 kernel: CPU: 4 PID: 1765 Comm: ptlrpcd_00_03 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:47 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:47 oak-gw06 kernel: 0000000000104020 000000009d48cbc7 ffff88043fd039f8 ffffffff8168662f Aug 14 20:22:47 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff88043fd03a08 0000000000000000 Aug 14 20:22:47 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88043ffd9000 000000009d48cbc7 Aug 14 20:22:47 oak-gw06 kernel: Call Trace: Aug 14 20:22:47 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:47 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:47 oak-gw06 kernel: [] ? __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:47 oak-gw06 kernel: [] ? __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:47 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:47 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] ? bnx2x_alloc_rx_data.isra.70+0x54/0x1c0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:47 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:47 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:47 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 14 20:22:47 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 14 20:22:47 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 14 20:22:47 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 14 20:22:47 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 14 20:22:47 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 14 20:22:47 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: ptlrpcd_00_03: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:47 oak-gw06 kernel: CPU: 4 PID: 1765 Comm: ptlrpcd_00_03 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:47 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:47 oak-gw06 kernel: 0000000000104020 000000009d48cbc7 ffff88043fd039f8 ffffffff8168662f Aug 14 20:22:47 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff88043fd03a08 0000000000000000 Aug 14 20:22:47 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88043ffd9000 000000009d48cbc7 Aug 14 20:22:47 oak-gw06 kernel: Call Trace: Aug 14 20:22:47 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:47 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:47 oak-gw06 kernel: [] ? __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:47 oak-gw06 kernel: [] ? __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:47 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:47 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] ? bnx2x_alloc_rx_data.isra.70+0x54/0x1c0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:47 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:47 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:47 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 14 20:22:47 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 14 20:22:47 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 14 20:22:47 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 14 20:22:47 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 14 20:22:47 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 14 20:22:47 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: ptlrpcd_00_03: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:47 oak-gw06 kernel: CPU: 4 PID: 1765 Comm: ptlrpcd_00_03 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:47 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:47 oak-gw06 kernel: 0000000000104020 000000009d48cbc7 ffff88043fd039f8 ffffffff8168662f Aug 14 20:22:47 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff88043fd03a08 0000000000000000 Aug 14 20:22:47 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88043ffd9000 000000009d48cbc7 Aug 14 20:22:47 oak-gw06 kernel: Call Trace: Aug 14 20:22:47 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:47 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:47 oak-gw06 kernel: [] ? __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:47 oak-gw06 kernel: [] ? __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:47 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:47 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] ? bnx2x_alloc_rx_data.isra.70+0x54/0x1c0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:47 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:47 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:47 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 14 20:22:47 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 14 20:22:47 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 14 20:22:47 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 14 20:22:47 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 14 20:22:47 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 14 20:22:47 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: ptlrpcd_00_03: page allocation failure: order:2, mode:0x104020 Aug 14 20:22:47 oak-gw06 kernel: CPU: 4 PID: 1765 Comm: ptlrpcd_00_03 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:47 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:47 oak-gw06 kernel: 0000000000104020 000000009d48cbc7 ffff88043fd039f8 ffffffff8168662f Aug 14 20:22:47 oak-gw06 kernel: ffff88043fd03a88 ffffffff81186ba0 ffff88043fd03a08 0000000000000000 Aug 14 20:22:47 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88043ffd9000 000000009d48cbc7 Aug 14 20:22:47 oak-gw06 kernel: Call Trace: Aug 14 20:22:47 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:47 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:47 oak-gw06 kernel: [] ? __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:47 oak-gw06 kernel: [] ? __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:47 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:47 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] ? bnx2x_alloc_rx_data.isra.70+0x54/0x1c0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:47 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:47 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:47 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] smp_apic_timer_interrupt+0x45/0x60 Aug 14 20:22:47 oak-gw06 kernel: [] apic_timer_interrupt+0x6d/0x80 Aug 14 20:22:47 oak-gw06 kernel: [] ? 0xffffffffa050604f Aug 14 20:22:47 oak-gw06 kernel: [] ? crc32_pclmul_le+0x5a/0x100 [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crc32_pclmul_update+0x17/0x1f [crc32_pclmul] Aug 14 20:22:47 oak-gw06 kernel: [] crypto_shash_update+0x47/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] ? crypto_shash_setkey+0x34/0xc0 Aug 14 20:22:47 oak-gw06 kernel: [] shash_ahash_update+0x3e/0x70 Aug 14 20:22:47 oak-gw06 kernel: [] shash_async_update+0x12/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] cfs_crypto_hash_update_page+0x7e/0xb0 [libcfs] Aug 14 20:22:47 oak-gw06 kernel: [] osc_checksum_bulk+0x11d/0x440 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] osc_brw_fini_request+0x1e7/0x12f0 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? at_measured+0x1c7/0x380 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] brw_interpret+0x57/0xe60 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ? __switch_to+0xd9/0x4c0 Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:22:47 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:22:47 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:22:47 oak-gw06 kernel: CPU: 3 PID: 3872 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:22:47 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:22:47 oak-gw06 kernel: 0000000000104020 00000000409aab66 ffff88043fcc39d8 ffffffff8168662f Aug 14 20:22:47 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 000068fc2a7fd630 0000000000000001 Aug 14 20:22:47 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff880023409740 00000000409aab66 Aug 14 20:22:47 oak-gw06 kernel: Call Trace: Aug 14 20:22:47 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:22:47 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:22:47 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:22:47 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:22:47 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:22:47 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:22:47 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:22:47 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:22:47 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:22:47 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:22:47 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:22:47 oak-gw06 kernel: [] ? __mem_cgroup_commit_charge+0x30/0x370 Aug 14 20:22:47 oak-gw06 kernel: [] ? __alloc_pages_nodemask+0x405/0x420 Aug 14 20:22:47 oak-gw06 kernel: [] mem_cgroup_charge_common+0x77/0xc0 Aug 14 20:22:47 oak-gw06 kernel: [] mem_cgroup_cache_charge+0x8a/0xb0 Aug 14 20:22:47 oak-gw06 kernel: [] __add_to_page_cache_locked+0x52/0x2b0 Aug 14 20:22:47 oak-gw06 kernel: [] add_to_page_cache_lru+0x37/0xb0 Aug 14 20:22:47 oak-gw06 kernel: [] grab_cache_page_nowait+0x49/0xa0 Aug 14 20:22:47 oak-gw06 kernel: [] ll_readahead+0xcbd/0x1660 [lustre] Aug 14 20:22:47 oak-gw06 kernel: [] ? ldlm_lock_decref+0x36/0x80 [ptlrpc] Aug 14 20:22:47 oak-gw06 kernel: [] ? osc_io_fini+0x10/0x10 [osc] Aug 14 20:22:47 oak-gw06 kernel: [] ll_readpage+0x3ba/0x6c0 [lustre] Aug 14 20:22:47 oak-gw06 kernel: [] generic_file_aio_read+0x3cf/0x790 Aug 14 20:22:47 oak-gw06 kernel: [] vvp_io_read_start+0x4b4/0x5a0 [lustre] Aug 14 20:22:47 oak-gw06 kernel: [] cl_io_start+0x65/0x130 [obdclass] Aug 14 20:22:47 oak-gw06 kernel: [] cl_io_loop+0xa5/0x190 [obdclass] Aug 14 20:22:47 oak-gw06 kernel: [] ll_file_io_generic+0x570/0xb50 [lustre] Aug 14 20:22:47 oak-gw06 kernel: [] ll_file_aio_read+0x347/0x3e0 [lustre] Aug 14 20:22:47 oak-gw06 kernel: [] ll_file_read+0xdb/0x3d0 [lustre] Aug 14 20:22:47 oak-gw06 kernel: [] vfs_read+0x9e/0x170 Aug 14 20:22:47 oak-gw06 kernel: [] SyS_read+0x7f/0xe0 Aug 14 20:22:47 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 14 20:23:00 oak-gw06 kernel: warn_alloc_failed: 488 callbacks suppressed Aug 14 20:23:00 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 20:23:00 oak-gw06 kernel: CPU: 6 PID: 3633 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:23:00 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:23:00 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:23:00 oak-gw06 kernel: 00000000000080d0 00000000f47ca210 ffff880007fa3858 ffffffff8168662f Aug 14 20:23:00 oak-gw06 kernel: ffff880007fa38e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 14 20:23:00 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff880007fa38e8 00000000f47ca210 Aug 14 20:23:00 oak-gw06 kernel: Call Trace: Aug 14 20:23:00 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:23:00 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:23:00 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:23:00 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:23:00 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 20:23:00 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 20:23:00 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:23:00 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:23:00 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:23:00 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:23:00 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:23:00 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:23:00 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:23:00 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:23:00 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:23:00 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:23:00 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:23:00 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:23:00 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:23:00 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:23:00 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:23:00 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:23:00 oak-gw06 kernel: Mem-Info: Aug 14 20:23:00 oak-gw06 kernel: active_anon:20481 inactive_anon:51096 isolated_anon:0#012 active_file:632501 inactive_file:1856607 isolated_file:0#012 unevictable:0 dirty:3168 writeback:2436 unstable:0#012 slab_reclaimable:41900 slab_unreclaimable:1021717#012 mapped:9558 shmem:45078 pagetables:1678 bounce:0#012 free:346728 free_pcp:1464 free_cma:0 Aug 14 20:23:00 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:23:00 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:23:00 oak-gw06 kernel: Node 0 DMA32 free:462472kB min:69724kB low:87152kB high:104584kB active_anon:11840kB inactive_anon:35588kB active_file:494092kB inactive_file:1295720kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3124kB writeback:7092kB mapped:4460kB shmem:31268kB slab_reclaimable:24464kB slab_unreclaimable:502760kB kernel_stack:960kB pagetables:1264kB unstable:0kB bounce:0kB free_pcp:2092kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:23:00 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:23:00 oak-gw06 kernel: Node 0 Normal free:957520kB min:323104kB low:403880kB high:484656kB active_anon:70084kB inactive_anon:168796kB active_file:2035912kB inactive_file:6084064kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8476kB writeback:6452kB mapped:33772kB shmem:149044kB slab_reclaimable:143136kB slab_unreclaimable:3584092kB kernel_stack:4736kB pagetables:5448kB unstable:0kB bounce:0kB free_pcp:3868kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:23:00 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:23:00 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:23:00 oak-gw06 kernel: Node 0 DMA32: 12315*4kB (UEM) 12024*8kB (UEM) 5533*16kB (UEM) 5499*32kB (UEM) 861*64kB (UM) 67*128kB (UM) 5*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 474908kB Aug 14 20:23:00 oak-gw06 kernel: Node 0 Normal: 69691*4kB (UEM) 42357*8kB (UEM) 13577*16kB (UEM) 4624*32kB (UEM) 313*64kB (UEM) 4*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1003364kB Aug 14 20:23:00 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:23:00 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:23:00 oak-gw06 kernel: 2112528 total pagecache pages Aug 14 20:23:00 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:23:00 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:23:00 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:23:00 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:23:00 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:23:00 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:23:00 oak-gw06 kernel: 127313 pages reserved Aug 14 20:23:00 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 20:23:00 oak-gw06 kernel: CPU: 6 PID: 3633 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:23:00 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:23:00 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:23:00 oak-gw06 kernel: 00000000000080d0 00000000f47ca210 ffff880007fa3808 ffffffff8168662f Aug 14 20:23:00 oak-gw06 kernel: ffff880007fa3898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 20:23:00 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880007fa3868 00000000f47ca210 Aug 14 20:23:00 oak-gw06 kernel: Call Trace: Aug 14 20:23:00 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:23:00 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:23:00 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 20:23:00 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:23:00 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:23:00 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:23:00 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:23:00 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 20:23:00 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 20:23:00 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:23:00 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:23:00 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:23:00 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:23:00 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:23:00 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:23:00 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:23:00 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:23:00 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:23:00 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:23:00 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:23:00 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:23:00 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:23:00 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:23:00 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:23:00 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:23:00 oak-gw06 kernel: Mem-Info: Aug 14 20:23:00 oak-gw06 kernel: active_anon:20481 inactive_anon:51096 isolated_anon:0#012 active_file:632375 inactive_file:1788285 isolated_file:0#012 unevictable:0 dirty:3069 writeback:2497 unstable:0#012 slab_reclaimable:41900 slab_unreclaimable:1021645#012 mapped:9558 shmem:45078 pagetables:1678 bounce:0#012 free:415581 free_pcp:1855 free_cma:0 Aug 14 20:23:00 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:23:00 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:23:00 oak-gw06 kernel: Node 0 DMA32 free:515132kB min:69724kB low:87152kB high:104584kB active_anon:11840kB inactive_anon:35588kB active_file:493588kB inactive_file:1249772kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2248kB writeback:1208kB mapped:4460kB shmem:31268kB slab_reclaimable:24464kB slab_unreclaimable:502760kB kernel_stack:960kB pagetables:1264kB unstable:0kB bounce:0kB free_pcp:3220kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:23:00 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:23:00 oak-gw06 kernel: Node 0 Normal free:1219572kB min:323104kB low:403880kB high:484656kB active_anon:70084kB inactive_anon:168796kB active_file:2035912kB inactive_file:5814184kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9640kB writeback:7228kB mapped:33772kB shmem:149044kB slab_reclaimable:143136kB slab_unreclaimable:3583804kB kernel_stack:4736kB pagetables:5448kB unstable:0kB bounce:0kB free_pcp:3896kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:23:00 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:23:00 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:23:00 oak-gw06 kernel: Node 0 DMA32: 16276*4kB (UEM) 14492*8kB (UEM) 6575*16kB (UEM) 5516*32kB (UEM) 863*64kB (UM) 67*128kB (UM) 5*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 527840kB Aug 14 20:23:00 oak-gw06 kernel: Node 0 Normal: 82860*4kB (UEM) 54798*8kB (UEM) 20297*16kB (UEM) 4999*32kB (UEM) 326*64kB (UEM) 4*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1275920kB Aug 14 20:23:00 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:23:00 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:23:00 oak-gw06 kernel: 2113025 total pagecache pages Aug 14 20:23:00 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:23:00 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:23:00 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:23:00 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:23:00 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:23:00 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:23:00 oak-gw06 kernel: 127313 pages reserved Aug 14 20:25:21 oak-gw06 kernel: globus-gridftp-: page allocation failure: order:2, mode:0x104020 Aug 14 20:25:21 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 14 20:25:21 oak-gw06 kernel: CPU: 3 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:25:21 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:25:21 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fcc39d8 ffffffff8168662f Aug 14 20:25:21 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 ffff88043fcc3a30 ffffffff81098f9b Aug 14 20:25:21 oak-gw06 kernel: fffffffffffffffc 0010402000000000 ffff88002340f440 00000000da86253b Aug 14 20:25:21 oak-gw06 kernel: Call Trace: Aug 14 20:25:21 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:25:21 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:25:21 oak-gw06 kernel: [] ? mod_timer+0x14b/0x230 Aug 14 20:25:21 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:25:21 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:25:21 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:25:21 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:25:21 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:25:21 oak-gw06 kernel: [] ? __dev_kfree_skb_any+0x3d/0x50 Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:25:21 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:25:21 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:25:21 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:25:21 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:25:21 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:25:21 oak-gw06 kernel: [] ? unfreeze_partials.isra.43+0xe7/0x130 Aug 14 20:25:21 oak-gw06 kernel: [] ? osc_teardown_async_page+0x177/0x5c0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 14 20:25:21 oak-gw06 kernel: [] ? cfs_hash_bd_from_key+0x32/0xb0 [libcfs] Aug 14 20:25:21 oak-gw06 kernel: [] ? lu_object_put+0x148/0x3d0 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] osc_page_delete+0x52/0x4e0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:25:21 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:25:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:25:21 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:25:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:25:21 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 14 20:25:21 oak-gw06 kernel: CPU: 3 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:25:21 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:25:21 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fcc39d8 ffffffff8168662f Aug 14 20:25:21 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 0000000000000010 0000000000000000 Aug 14 20:25:21 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000205220 00000000da86253b Aug 14 20:25:21 oak-gw06 kernel: Call Trace: Aug 14 20:25:21 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:25:21 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:25:21 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:25:21 oak-gw06 kernel: [] ? alloc_pages_current+0xa0/0x170 Aug 14 20:25:21 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:25:21 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:25:21 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:25:21 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:25:21 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:25:21 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:25:21 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:25:21 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:25:21 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:25:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:25:21 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:25:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:25:21 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 14 20:25:21 oak-gw06 kernel: CPU: 3 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:25:21 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:25:21 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fcc39d8 ffffffff8168662f Aug 14 20:25:21 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 0000000000000010 0000000000000000 Aug 14 20:25:21 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000205220 00000000da86253b Aug 14 20:25:21 oak-gw06 kernel: Call Trace: Aug 14 20:25:21 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:25:21 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:25:21 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:25:21 oak-gw06 kernel: [] ? alloc_pages_current+0xa0/0x170 Aug 14 20:25:21 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:25:21 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:25:21 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:25:21 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:25:21 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:25:21 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:25:21 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:25:21 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:25:21 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:25:21 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:25:21 oak-gw06 kernel: [] ? unfreeze_partials.isra.43+0xe7/0x130 Aug 14 20:25:21 oak-gw06 kernel: [] ? osc_teardown_async_page+0x177/0x5c0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 14 20:25:21 oak-gw06 kernel: [] ? cfs_hash_bd_from_key+0x32/0xb0 [libcfs] Aug 14 20:25:21 oak-gw06 kernel: [] ? lu_object_put+0x148/0x3d0 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] osc_page_delete+0x52/0x4e0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:25:21 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:25:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:25:21 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:25:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:25:21 oak-gw06 kernel: [] ? unfreeze_partials.isra.43+0xe7/0x130 Aug 14 20:25:21 oak-gw06 kernel: [] ? osc_teardown_async_page+0x177/0x5c0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 14 20:25:21 oak-gw06 kernel: [] ? cfs_hash_bd_from_key+0x32/0xb0 [libcfs] Aug 14 20:25:21 oak-gw06 kernel: [] ? lu_object_put+0x148/0x3d0 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] osc_page_delete+0x52/0x4e0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:25:21 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:25:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:25:21 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:25:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:25:21 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 14 20:25:21 oak-gw06 kernel: CPU: 3 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:25:21 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:25:21 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fcc39d8 ffffffff8168662f Aug 14 20:25:21 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 0000000000000010 0000000000000000 Aug 14 20:25:21 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000205220 00000000da86253b Aug 14 20:25:21 oak-gw06 kernel: Call Trace: Aug 14 20:25:21 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:25:21 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:25:21 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:25:21 oak-gw06 kernel: [] ? alloc_pages_current+0xa0/0x170 Aug 14 20:25:21 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:25:21 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:25:21 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:25:21 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:25:21 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:25:21 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:25:21 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:25:21 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:25:21 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:25:21 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:25:21 oak-gw06 kernel: [] ? unfreeze_partials.isra.43+0xe7/0x130 Aug 14 20:25:21 oak-gw06 kernel: [] ? osc_teardown_async_page+0x177/0x5c0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 14 20:25:21 oak-gw06 kernel: [] ? cfs_hash_bd_from_key+0x32/0xb0 [libcfs] Aug 14 20:25:21 oak-gw06 kernel: [] ? lu_object_put+0x148/0x3d0 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] osc_page_delete+0x52/0x4e0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:25:21 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:25:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:25:21 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:25:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:25:21 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 14 20:25:21 oak-gw06 kernel: CPU: 3 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:25:21 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:25:21 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fcc39d8 ffffffff8168662f Aug 14 20:25:21 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 0000000000000010 0000000000000000 Aug 14 20:25:21 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000205220 00000000da86253b Aug 14 20:25:21 oak-gw06 kernel: Call Trace: Aug 14 20:25:21 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:25:21 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:25:21 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:25:21 oak-gw06 kernel: [] ? alloc_pages_current+0xa0/0x170 Aug 14 20:25:21 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:25:21 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:25:21 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:25:21 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:25:21 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:25:21 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:25:21 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:25:21 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:25:21 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:25:21 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:25:21 oak-gw06 kernel: [] ? unfreeze_partials.isra.43+0xe7/0x130 Aug 14 20:25:21 oak-gw06 kernel: [] ? osc_teardown_async_page+0x177/0x5c0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 14 20:25:21 oak-gw06 kernel: [] ? cfs_hash_bd_from_key+0x32/0xb0 [libcfs] Aug 14 20:25:21 oak-gw06 kernel: [] ? lu_object_put+0x148/0x3d0 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] osc_page_delete+0x52/0x4e0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:25:21 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:25:21 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:25:21 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:25:21 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:25:21 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:25:21 oak-gw06 kernel: [] ? unfreeze_partials.isra.43+0xe7/0x130 Aug 14 20:25:21 oak-gw06 kernel: [] ? osc_teardown_async_page+0x177/0x5c0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 14 20:25:21 oak-gw06 kernel: [] ? cfs_hash_bd_from_key+0x32/0xb0 [libcfs] Aug 14 20:25:21 oak-gw06 kernel: [] ? lu_object_put+0x148/0x3d0 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] osc_page_delete+0x52/0x4e0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:25:21 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:25:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:25:21 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:25:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:25:21 oak-gw06 kernel: ptlrpcd_00_05: page allocation failure: order:2, mode:0x104020 Aug 14 20:25:21 oak-gw06 kernel: CPU: 3 PID: 1767 Comm: ptlrpcd_00_05 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:25:21 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:25:21 oak-gw06 kernel: 0000000000104020 00000000da86253b ffff88043fcc39d8 ffffffff8168662f Aug 14 20:25:21 oak-gw06 kernel: ffff88043fcc3a68 ffffffff81186ba0 0000000000000010 0000000000000000 Aug 14 20:25:21 oak-gw06 kernel: fffffffffffffffc 0010402000000000 0000000000205220 00000000da86253b Aug 14 20:25:21 oak-gw06 kernel: Call Trace: Aug 14 20:25:21 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:25:21 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:25:21 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:25:21 oak-gw06 kernel: [] ? alloc_pages_current+0xa0/0x170 Aug 14 20:25:21 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:25:21 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:25:21 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:25:21 oak-gw06 kernel: [] kmalloc_order_trace+0x2e/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] ? __kmalloc+0x221/0x240 Aug 14 20:25:21 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:25:21 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:25:21 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:25:21 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] irq_exit+0x115/0x120 Aug 14 20:25:21 oak-gw06 kernel: [] do_IRQ+0x58/0xf0 Aug 14 20:25:21 oak-gw06 kernel: [] common_interrupt+0x6d/0x6d Aug 14 20:25:21 oak-gw06 kernel: [] ? unfreeze_partials.isra.43+0xe7/0x130 Aug 14 20:25:21 oak-gw06 kernel: [] ? osc_teardown_async_page+0x177/0x5c0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] ? __slab_free+0x81/0x2f0 Aug 14 20:25:21 oak-gw06 kernel: [] ? cfs_hash_bd_from_key+0x32/0xb0 [libcfs] Aug 14 20:25:21 oak-gw06 kernel: [] ? lu_object_put+0x148/0x3d0 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] osc_page_delete+0x52/0x4e0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] cl_page_delete0+0x7d/0x210 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] cl_page_delete+0x33/0x110 [obdclass] Aug 14 20:25:21 oak-gw06 kernel: [] discard_pagevec+0x52/0xd0 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] osc_lru_shrink+0x6da/0x750 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] lru_queue_work+0x4c/0x230 [osc] Aug 14 20:25:21 oak-gw06 kernel: [] work_interpreter+0x37/0xf0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpc_check_set.part.24+0x425/0x1dd0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ptlrpcd+0x227/0x560 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] ? wake_up_state+0x20/0x20 Aug 14 20:25:21 oak-gw06 kernel: [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] Aug 14 20:25:21 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:25:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:25:21 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:25:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:25:21 oak-gw06 kernel: CPU: 4 PID: 3874 Comm: globus-gridftp- Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:25:21 oak-gw06 kernel: [] __kmalloc+0x221/0x240 Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] ? consume_skb+0x34/0x80 Aug 14 20:25:21 oak-gw06 kernel: [] bnx2x_poll+0x1dd/0x260 [bnx2x] Aug 14 20:25:21 oak-gw06 kernel: [] net_rx_action+0x170/0x380 Aug 14 20:25:21 oak-gw06 kernel: [] __do_softirq+0xef/0x280 Aug 14 20:25:21 oak-gw06 kernel: [] call_softirq+0x1c/0x30 Aug 14 20:25:21 oak-gw06 kernel: [] do_softirq+0x65/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] local_bh_enable+0x94/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] lock_sock_nested+0x43/0x50 Aug 14 20:25:21 oak-gw06 kernel: [] tcp_recvmsg+0x91/0xb50 Aug 14 20:25:21 oak-gw06 kernel: [] ? poll_select_copy_remaining+0x150/0x150 Aug 14 20:25:21 oak-gw06 kernel: [] ? poll_select_copy_remaining+0x150/0x150 Aug 14 20:25:21 oak-gw06 kernel: [] inet_recvmsg+0x7b/0xa0 Aug 14 20:25:21 oak-gw06 kernel: [] sock_recvmsg+0xbf/0x100 Aug 14 20:25:21 oak-gw06 kernel: [] SYSC_recvfrom+0xe8/0x160 Aug 14 20:25:21 oak-gw06 kernel: [] ? poll_select_copy_remaining+0xfc/0x150 Aug 14 20:25:21 oak-gw06 kernel: [] SyS_recvfrom+0xe/0x10 Aug 14 20:25:21 oak-gw06 kernel: [] system_call_fastpath+0x16/0x1b Aug 14 20:28:00 oak-gw06 kernel: warn_alloc_failed: 62 callbacks suppressed Aug 14 20:28:00 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 20:28:00 oak-gw06 kernel: CPU: 6 PID: 3633 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:28:00 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:28:00 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:28:00 oak-gw06 kernel: 00000000000080d0 00000000f47ca210 ffff880007fa3858 ffffffff8168662f Aug 14 20:28:00 oak-gw06 kernel: ffff880007fa38e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 20:28:00 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880007fa38b8 00000000f47ca210 Aug 14 20:28:00 oak-gw06 kernel: Call Trace: Aug 14 20:28:00 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:28:00 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:28:00 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:28:00 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:28:00 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 20:28:00 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 20:28:00 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:28:00 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:28:00 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:28:00 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:28:00 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:28:00 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:28:00 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:28:00 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:28:00 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:28:00 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:28:00 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:28:00 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:28:00 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:28:00 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:28:00 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:28:00 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:28:00 oak-gw06 kernel: Mem-Info: Aug 14 20:28:00 oak-gw06 kernel: active_anon:24440 inactive_anon:51096 isolated_anon:0#012 active_file:506171 inactive_file:1965810 isolated_file:0#012 unevictable:0 dirty:15203 writeback:4336 unstable:0#012 slab_reclaimable:39772 slab_unreclaimable:946062#012 mapped:9939 shmem:45078 pagetables:1683 bounce:0#012 free:407108 free_pcp:101 free_cma:0 Aug 14 20:28:00 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:28:00 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:28:00 oak-gw06 kernel: Node 0 DMA32 free:419120kB min:69724kB low:87152kB high:104584kB active_anon:16256kB inactive_anon:35588kB active_file:509192kB inactive_file:1342660kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:10816kB writeback:3316kB mapped:4944kB shmem:31268kB slab_reclaimable:23100kB slab_unreclaimable:463408kB kernel_stack:976kB pagetables:1092kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:28:00 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:28:00 oak-gw06 kernel: Node 0 Normal free:1175896kB min:323104kB low:403880kB high:484656kB active_anon:81504kB inactive_anon:168796kB active_file:1515492kB inactive_file:6536960kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:49996kB writeback:14028kB mapped:34812kB shmem:149044kB slab_reclaimable:135988kB slab_unreclaimable:3320824kB kernel_stack:4720kB pagetables:5640kB unstable:0kB bounce:0kB free_pcp:432kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:28:00 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:28:00 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:28:00 oak-gw06 kernel: Node 0 DMA32: 6665*4kB (UEM) 6772*8kB (UEM) 11076*16kB (UEM) 4107*32kB (UEM) 484*64kB (UM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 420580kB Aug 14 20:28:00 oak-gw06 kernel: Node 0 Normal: 37931*4kB (UE) 35750*8kB (UE) 35794*16kB (UEM) 4328*32kB (UEM) 246*64kB (UM) 4*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1165180kB Aug 14 20:28:00 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:28:00 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:28:00 oak-gw06 kernel: 1987484 total pagecache pages Aug 14 20:28:00 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:28:00 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:28:00 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:28:00 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:28:00 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:28:00 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:28:00 oak-gw06 kernel: 127313 pages reserved Aug 14 20:28:00 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 20:28:00 oak-gw06 kernel: CPU: 6 PID: 3633 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:28:00 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:28:00 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:28:00 oak-gw06 kernel: 00000000000080d0 00000000f47ca210 ffff880007fa3808 ffffffff8168662f Aug 14 20:28:00 oak-gw06 kernel: ffff880007fa3898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 20:28:00 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880007fa3868 00000000f47ca210 Aug 14 20:28:00 oak-gw06 kernel: Call Trace: Aug 14 20:28:00 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:28:00 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:28:00 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 20:28:00 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:28:00 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:28:00 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:28:00 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:28:00 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 20:28:00 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 20:28:00 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:28:00 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:28:00 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:28:00 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:28:00 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:28:00 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:28:00 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:28:00 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:28:00 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:28:00 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:28:00 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:28:00 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:28:00 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:28:00 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:28:00 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:28:00 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:28:00 oak-gw06 kernel: Mem-Info: Aug 14 20:28:00 oak-gw06 kernel: active_anon:24440 inactive_anon:51096 isolated_anon:0#012 active_file:506171 inactive_file:1979395 isolated_file:0#012 unevictable:0 dirty:15203 writeback:4336 unstable:0#012 slab_reclaimable:39772 slab_unreclaimable:946062#012 mapped:9939 shmem:45078 pagetables:1683 bounce:0#012 free:391749 free_pcp:177 free_cma:0 Aug 14 20:28:00 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:28:00 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:28:00 oak-gw06 kernel: Node 0 DMA32 free:419408kB min:69724kB low:87152kB high:104584kB active_anon:16256kB inactive_anon:35588kB active_file:509192kB inactive_file:1342660kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:10816kB writeback:3316kB mapped:4944kB shmem:31268kB slab_reclaimable:23100kB slab_unreclaimable:463408kB kernel_stack:976kB pagetables:1092kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:28:00 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:28:00 oak-gw06 kernel: Node 0 Normal free:1126048kB min:323104kB low:403880kB high:484656kB active_anon:81504kB inactive_anon:168796kB active_file:1515492kB inactive_file:6578300kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:49996kB writeback:14028kB mapped:34812kB shmem:149044kB slab_reclaimable:135988kB slab_unreclaimable:3320824kB kernel_stack:4720kB pagetables:5640kB unstable:0kB bounce:0kB free_pcp:1044kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:28:00 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:28:00 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:28:00 oak-gw06 kernel: Node 0 DMA32: 6666*4kB (UEM) 6772*8kB (UEM) 11076*16kB (UEM) 4107*32kB (UEM) 484*64kB (UM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 420584kB Aug 14 20:28:00 oak-gw06 kernel: Node 0 Normal: 37069*4kB (UEM) 35754*8kB (UEM) 33468*16kB (UEM) 4328*32kB (UEM) 246*64kB (UM) 4*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1124548kB Aug 14 20:28:00 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:28:00 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:28:00 oak-gw06 kernel: 1996796 total pagecache pages Aug 14 20:28:00 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:28:00 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:28:00 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:28:00 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:28:00 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:28:00 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:28:00 oak-gw06 kernel: 127313 pages reserved Aug 14 20:33:01 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 20:33:01 oak-gw06 kernel: CPU: 3 PID: 3699 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:33:01 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:33:01 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:33:01 oak-gw06 kernel: 00000000000080d0 0000000084406979 ffff88018901b858 ffffffff8168662f Aug 14 20:33:01 oak-gw06 kernel: ffff88018901b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 20:33:01 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88018901b8b8 0000000084406979 Aug 14 20:33:01 oak-gw06 kernel: Call Trace: Aug 14 20:33:01 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:33:01 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:33:01 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:33:01 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:33:01 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 20:33:01 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 20:33:01 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:33:01 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:33:01 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:33:01 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:33:01 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:33:01 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:33:01 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:33:01 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:33:01 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:33:01 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:33:01 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:33:01 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:33:01 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:33:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:33:01 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:33:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:33:01 oak-gw06 kernel: Mem-Info: Aug 14 20:33:01 oak-gw06 kernel: active_anon:24815 inactive_anon:51096 isolated_anon:0#012 active_file:668087 inactive_file:1624583 isolated_file:0#012 unevictable:0 dirty:2194 writeback:1878 unstable:0#012 slab_reclaimable:39721 slab_unreclaimable:937624#012 mapped:10213 shmem:45078 pagetables:1686 bounce:0#012 free:596836 free_pcp:869 free_cma:0 Aug 14 20:33:01 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:33:01 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:33:01 oak-gw06 kernel: Node 0 DMA32 free:551784kB min:69724kB low:87152kB high:104584kB active_anon:15324kB inactive_anon:35588kB active_file:534996kB inactive_file:1179160kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1156kB writeback:1292kB mapped:4708kB shmem:31268kB slab_reclaimable:23084kB slab_unreclaimable:464992kB kernel_stack:960kB pagetables:1092kB unstable:0kB bounce:0kB free_pcp:1128kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:33:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:33:01 oak-gw06 kernel: Node 0 Normal free:1821888kB min:323104kB low:403880kB high:484656kB active_anon:84196kB inactive_anon:168796kB active_file:2140732kB inactive_file:5325576kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8480kB writeback:9324kB mapped:36144kB shmem:149044kB slab_reclaimable:135800kB slab_unreclaimable:3286864kB kernel_stack:4736kB pagetables:5652kB unstable:0kB bounce:0kB free_pcp:2572kB local_pcp:60kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:33:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:33:01 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:33:01 oak-gw06 kernel: Node 0 DMA32: 7199*4kB (UEM) 13760*8kB (UEM) 13961*16kB (UEM) 3874*32kB (UEM) 998*64kB (UM) 25*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 553292kB Aug 14 20:33:01 oak-gw06 kernel: Node 0 Normal: 43862*4kB (UEM) 62639*8kB (UEM) 49593*16kB (UEM) 10221*32kB (UEM) 303*64kB (UM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1816640kB Aug 14 20:33:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:33:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:33:01 oak-gw06 kernel: 2062644 total pagecache pages Aug 14 20:33:01 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:33:01 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:33:01 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:33:01 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:33:01 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:33:01 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:33:01 oak-gw06 kernel: 127313 pages reserved Aug 14 20:33:01 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 20:33:01 oak-gw06 kernel: CPU: 3 PID: 3699 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:33:01 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:33:01 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:33:01 oak-gw06 kernel: 00000000000080d0 0000000084406979 ffff88018901b808 ffffffff8168662f Aug 14 20:33:01 oak-gw06 kernel: ffff88018901b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 20:33:01 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88018901b868 0000000084406979 Aug 14 20:33:01 oak-gw06 kernel: Call Trace: Aug 14 20:33:01 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:33:01 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:33:01 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:33:01 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:33:01 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:33:01 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:33:01 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 20:33:01 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 20:33:01 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:33:01 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:33:01 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:33:01 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:33:01 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:33:01 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:33:01 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:33:01 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:33:01 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:33:01 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:33:01 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:33:01 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:33:01 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:33:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:33:01 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:33:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:33:01 oak-gw06 kernel: Mem-Info: Aug 14 20:33:01 oak-gw06 kernel: active_anon:24815 inactive_anon:51096 isolated_anon:0#012 active_file:671520 inactive_file:1630876 isolated_file:0#012 unevictable:0 dirty:2409 writeback:2078 unstable:0#012 slab_reclaimable:39721 slab_unreclaimable:938700#012 mapped:10213 shmem:45078 pagetables:1686 bounce:0#012 free:593646 free_pcp:914 free_cma:0 Aug 14 20:33:01 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:33:01 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:33:01 oak-gw06 kernel: Node 0 DMA32 free:562100kB min:69724kB low:87152kB high:104584kB active_anon:15324kB inactive_anon:35588kB active_file:536508kB inactive_file:1177648kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1156kB writeback:540kB mapped:4708kB shmem:31268kB slab_reclaimable:23084kB slab_unreclaimable:464992kB kernel_stack:960kB pagetables:1092kB unstable:0kB bounce:0kB free_pcp:1572kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:33:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:33:01 oak-gw06 kernel: Node 0 Normal free:1787620kB min:323104kB low:403880kB high:484656kB active_anon:83936kB inactive_anon:168796kB active_file:2151652kB inactive_file:5356516kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8480kB writeback:5056kB mapped:36144kB shmem:149044kB slab_reclaimable:135800kB slab_unreclaimable:3291672kB kernel_stack:4736kB pagetables:5652kB unstable:0kB bounce:0kB free_pcp:2128kB local_pcp:52kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:33:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:33:01 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:33:01 oak-gw06 kernel: Node 0 DMA32: 8296*4kB (UE) 13719*8kB (UEM) 14188*16kB (UEM) 3969*32kB (UEM) 998*64kB (UM) 25*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 564024kB Aug 14 20:33:01 oak-gw06 kernel: Node 0 Normal: 45033*4kB (UEM) 57418*8kB (UEM) 49339*16kB (UEM) 10300*32kB (UEM) 308*64kB (UM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1778340kB Aug 14 20:33:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:33:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:33:01 oak-gw06 kernel: 2073743 total pagecache pages Aug 14 20:33:01 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:33:01 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:33:01 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:33:01 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:33:01 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:33:01 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:33:01 oak-gw06 kernel: 127313 pages reserved Aug 14 20:38:01 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 20:38:01 oak-gw06 kernel: CPU: 6 PID: 3893 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:38:01 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:38:01 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:38:01 oak-gw06 kernel: 00000000000080d0 00000000edecd00c ffff88019d533858 ffffffff8168662f Aug 14 20:38:01 oak-gw06 kernel: ffff88019d5338e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 20:38:01 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019d5338b8 00000000edecd00c Aug 14 20:38:01 oak-gw06 kernel: Call Trace: Aug 14 20:38:01 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:38:01 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:38:01 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:38:01 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:38:01 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 20:38:01 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 20:38:01 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:38:01 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:38:01 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:38:01 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:38:01 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:38:01 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:38:01 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:38:01 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:38:01 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:38:01 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:38:01 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:38:01 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:38:01 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:38:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:38:01 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:38:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:38:01 oak-gw06 kernel: Mem-Info: Aug 14 20:38:01 oak-gw06 kernel: active_anon:24802 inactive_anon:51096 isolated_anon:0#012 active_file:654112 inactive_file:1539777 isolated_file:0#012 unevictable:0 dirty:1883 writeback:706 unstable:0#012 slab_reclaimable:39681 slab_unreclaimable:905050#012 mapped:9938 shmem:45078 pagetables:1700 bounce:0#012 free:729139 free_pcp:595 free_cma:0 Aug 14 20:38:01 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:38:01 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:38:01 oak-gw06 kernel: Node 0 DMA32 free:688784kB min:69724kB low:87152kB high:104584kB active_anon:14680kB inactive_anon:35588kB active_file:527532kB inactive_file:1073224kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:660kB writeback:0kB mapped:4588kB shmem:31268kB slab_reclaimable:23084kB slab_unreclaimable:444672kB kernel_stack:944kB pagetables:1088kB unstable:0kB bounce:0kB free_pcp:1368kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:38:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:38:01 oak-gw06 kernel: Node 0 Normal free:2206072kB min:323104kB low:403880kB high:484656kB active_anon:84528kB inactive_anon:168796kB active_file:2100356kB inactive_file:5079644kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:6872kB writeback:1272kB mapped:35164kB shmem:149044kB slab_reclaimable:135640kB slab_unreclaimable:3175512kB kernel_stack:4752kB pagetables:5712kB unstable:0kB bounce:0kB free_pcp:1428kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:38:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:38:01 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:38:01 oak-gw06 kernel: Node 0 DMA32: 6351*4kB (UEM) 14760*8kB (UEM) 16680*16kB (UEM) 6209*32kB (UEM) 1208*64kB (UM) 41*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 691612kB Aug 14 20:38:01 oak-gw06 kernel: Node 0 Normal: 42677*4kB (UEM) 73290*8kB (UEM) 53952*16kB (UEM) 16350*32kB (UEM) 895*64kB (UEM) 10*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2202020kB Aug 14 20:38:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:38:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:38:01 oak-gw06 kernel: 1898296 total pagecache pages Aug 14 20:38:01 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:38:01 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:38:01 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:38:01 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:38:01 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:38:01 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:38:01 oak-gw06 kernel: 127313 pages reserved Aug 14 20:38:01 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 20:38:01 oak-gw06 kernel: CPU: 6 PID: 3893 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:38:01 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:38:01 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:38:01 oak-gw06 kernel: 00000000000080d0 00000000edecd00c ffff88019d533808 ffffffff8168662f Aug 14 20:38:01 oak-gw06 kernel: ffff88019d533898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 20:38:01 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019d533868 00000000edecd00c Aug 14 20:38:01 oak-gw06 kernel: Call Trace: Aug 14 20:38:01 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:38:01 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:38:01 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 20:38:01 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:38:01 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:38:01 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:38:01 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:38:01 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 20:38:01 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 20:38:01 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:38:01 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:38:01 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:38:01 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:38:01 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:38:01 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:38:01 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:38:01 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:38:01 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:38:01 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:38:01 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:38:01 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:38:01 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:38:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:38:01 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:38:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:38:01 oak-gw06 kernel: Mem-Info: Aug 14 20:38:01 oak-gw06 kernel: active_anon:24802 inactive_anon:51096 isolated_anon:0#012 active_file:663932 inactive_file:1534572 isolated_file:0#012 unevictable:0 dirty:1980 writeback:997 unstable:0#012 slab_reclaimable:39681 slab_unreclaimable:904984#012 mapped:9938 shmem:45078 pagetables:1700 bounce:0#012 free:724940 free_pcp:521 free_cma:0 Aug 14 20:38:01 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:38:01 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:38:01 oak-gw06 kernel: Node 0 DMA32 free:694328kB min:69724kB low:87152kB high:104584kB active_anon:14680kB inactive_anon:35588kB active_file:532964kB inactive_file:1067792kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:660kB writeback:0kB mapped:4588kB shmem:31268kB slab_reclaimable:23084kB slab_unreclaimable:444672kB kernel_stack:944kB pagetables:1088kB unstable:0kB bounce:0kB free_pcp:896kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:38:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:38:01 oak-gw06 kernel: Node 0 Normal free:2182440kB min:323104kB low:403880kB high:484656kB active_anon:84788kB inactive_anon:168796kB active_file:2131296kB inactive_file:5066904kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:6872kB writeback:2048kB mapped:35164kB shmem:149044kB slab_reclaimable:135640kB slab_unreclaimable:3175248kB kernel_stack:4752kB pagetables:5712kB unstable:0kB bounce:0kB free_pcp:1544kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:38:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:38:01 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:38:01 oak-gw06 kernel: Node 0 DMA32: 6986*4kB (UEM) 14763*8kB (UEM) 16998*16kB (UEM) 6227*32kB (UEM) 1208*64kB (UM) 41*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 699840kB Aug 14 20:38:01 oak-gw06 kernel: Node 0 Normal: 42273*4kB (UE) 71078*8kB (UEM) 53606*16kB (UEM) 16361*32kB (UEM) 895*64kB (UEM) 10*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2177524kB Aug 14 20:38:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:38:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:38:01 oak-gw06 kernel: 1902758 total pagecache pages Aug 14 20:38:01 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:38:01 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:38:01 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:38:01 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:38:01 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:38:01 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:38:01 oak-gw06 kernel: 127313 pages reserved Aug 14 20:43:01 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 20:43:01 oak-gw06 kernel: CPU: 6 PID: 3699 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:43:01 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:43:01 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:43:01 oak-gw06 kernel: 00000000000080d0 0000000084406979 ffff88018901b858 ffffffff8168662f Aug 14 20:43:01 oak-gw06 kernel: ffff88018901b8e8 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Aug 14 20:43:01 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 0000000084406979 Aug 14 20:43:01 oak-gw06 kernel: Call Trace: Aug 14 20:43:01 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:43:01 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:43:01 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Aug 14 20:43:01 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:43:01 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:43:01 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 20:43:01 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 20:43:01 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:43:01 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:43:01 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:43:01 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:43:01 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:43:01 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:43:01 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:43:01 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:43:01 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:43:01 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:43:01 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:43:01 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:43:01 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:43:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:43:01 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:43:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:43:01 oak-gw06 kernel: Mem-Info: Aug 14 20:43:01 oak-gw06 kernel: active_anon:24430 inactive_anon:51096 isolated_anon:0#012 active_file:968195 inactive_file:1341734 isolated_file:0#012 unevictable:0 dirty:2464 writeback:599 unstable:0#012 slab_reclaimable:39623 slab_unreclaimable:912504#012 mapped:9942 shmem:45078 pagetables:1700 bounce:0#012 free:632684 free_pcp:565 free_cma:0 Aug 14 20:43:01 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:43:01 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:43:01 oak-gw06 kernel: Node 0 DMA32 free:519416kB min:69724kB low:87152kB high:104584kB active_anon:10512kB inactive_anon:35588kB active_file:746816kB inactive_file:1054112kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1700kB writeback:176kB mapped:4592kB shmem:31268kB slab_reclaimable:23080kB slab_unreclaimable:442168kB kernel_stack:944kB pagetables:1084kB unstable:0kB bounce:0kB free_pcp:496kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:43:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:43:01 oak-gw06 kernel: Node 0 Normal free:1984540kB min:323104kB low:403880kB high:484656kB active_anon:87468kB inactive_anon:168796kB active_file:3125964kB inactive_file:4321664kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:6992kB writeback:4936kB mapped:35176kB shmem:149044kB slab_reclaimable:135412kB slab_unreclaimable:3207832kB kernel_stack:4768kB pagetables:5716kB unstable:0kB bounce:0kB free_pcp:1952kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:43:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:43:01 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:43:01 oak-gw06 kernel: Node 0 DMA32: 10140*4kB (UEM) 8545*8kB (UEM) 5562*16kB (UEM) 6320*32kB (UEM) 1751*64kB (UM) 74*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 521688kB Aug 14 20:43:01 oak-gw06 kernel: Node 0 Normal: 58112*4kB (UE) 45912*8kB (UE) 33874*16kB (UEM) 20611*32kB (UEM) 2564*64kB (UM) 32*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1969472kB Aug 14 20:43:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:43:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:43:01 oak-gw06 kernel: 2034108 total pagecache pages Aug 14 20:43:01 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:43:01 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:43:01 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:43:01 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:43:01 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:43:01 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:43:01 oak-gw06 kernel: 127313 pages reserved Aug 14 20:43:01 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 20:43:01 oak-gw06 kernel: CPU: 6 PID: 3699 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:43:01 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:43:01 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:43:01 oak-gw06 kernel: 00000000000080d0 0000000084406979 ffff88018901b808 ffffffff8168662f Aug 14 20:43:01 oak-gw06 kernel: ffff88018901b898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 20:43:01 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88018901b868 0000000084406979 Aug 14 20:43:01 oak-gw06 kernel: Call Trace: Aug 14 20:43:01 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:43:01 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:43:01 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 20:43:01 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:43:01 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:43:01 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:43:01 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:43:01 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 20:43:01 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 20:43:01 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:43:01 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:43:01 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:43:01 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:43:01 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:43:01 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:43:01 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:43:01 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:43:01 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:43:01 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:43:01 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:43:01 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:43:01 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:43:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:43:01 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:43:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:43:01 oak-gw06 kernel: Mem-Info: Aug 14 20:43:01 oak-gw06 kernel: active_anon:24430 inactive_anon:51096 isolated_anon:0#012 active_file:968195 inactive_file:1352488 isolated_file:0#012 unevictable:0 dirty:2451 writeback:1183 unstable:0#012 slab_reclaimable:39623 slab_unreclaimable:912880#012 mapped:9942 shmem:45078 pagetables:1700 bounce:0#012 free:620580 free_pcp:723 free_cma:0 Aug 14 20:43:01 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:43:01 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:43:01 oak-gw06 kernel: Node 0 DMA32 free:508764kB min:69724kB low:87152kB high:104584kB active_anon:10512kB inactive_anon:35588kB active_file:746816kB inactive_file:1062548kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2036kB writeback:960kB mapped:4592kB shmem:31268kB slab_reclaimable:23080kB slab_unreclaimable:442584kB kernel_stack:944kB pagetables:1084kB unstable:0kB bounce:0kB free_pcp:628kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:43:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:43:01 oak-gw06 kernel: Node 0 Normal free:1939668kB min:323104kB low:403880kB high:484656kB active_anon:87208kB inactive_anon:168796kB active_file:3125964kB inactive_file:4362744kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8156kB writeback:4936kB mapped:35176kB shmem:149044kB slab_reclaimable:135412kB slab_unreclaimable:3211096kB kernel_stack:4768kB pagetables:5716kB unstable:0kB bounce:0kB free_pcp:2544kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:43:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:43:01 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:43:01 oak-gw06 kernel: Node 0 DMA32: 9871*4kB (UEM) 8398*8kB (UEM) 4964*16kB (UEM) 6323*32kB (UEM) 1751*64kB (UM) 74*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 509964kB Aug 14 20:43:01 oak-gw06 kernel: Node 0 Normal: 57960*4kB (UEM) 45913*8kB (UEM) 31270*16kB (UEM) 20627*32kB (UEM) 2564*64kB (UM) 32*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1927720kB Aug 14 20:43:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:43:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:43:01 oak-gw06 kernel: 2043142 total pagecache pages Aug 14 20:43:01 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:43:01 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:43:01 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:43:01 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:43:01 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:43:01 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:43:01 oak-gw06 kernel: 127313 pages reserved Aug 14 20:48:01 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 20:48:01 oak-gw06 kernel: CPU: 6 PID: 3905 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:48:01 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:48:01 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:48:01 oak-gw06 kernel: 00000000000080d0 0000000068f894c0 ffff880123193858 ffffffff8168662f Aug 14 20:48:01 oak-gw06 kernel: ffff8801231938e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 20:48:01 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801231938b8 0000000068f894c0 Aug 14 20:48:01 oak-gw06 kernel: Call Trace: Aug 14 20:48:01 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:48:01 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:48:01 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:48:01 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:48:01 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 20:48:01 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 20:48:01 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:48:01 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:48:01 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:48:01 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:48:01 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:48:01 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:48:01 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:48:01 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:48:01 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:48:01 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:48:01 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:48:01 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:48:01 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:48:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:48:01 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:48:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:48:01 oak-gw06 kernel: Mem-Info: Aug 14 20:48:01 oak-gw06 kernel: active_anon:20438 inactive_anon:51096 isolated_anon:0#012 active_file:568970 inactive_file:1529027 isolated_file:0#012 unevictable:0 dirty:2209 writeback:1037 unstable:0#012 slab_reclaimable:39607 slab_unreclaimable:912081#012 mapped:9947 shmem:45078 pagetables:1681 bounce:0#012 free:850834 free_pcp:1018 free_cma:0 Aug 14 20:48:01 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:48:01 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:48:01 oak-gw06 kernel: Node 0 DMA32 free:677388kB min:69724kB low:87152kB high:104584kB active_anon:10592kB inactive_anon:35588kB active_file:450084kB inactive_file:1175632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2064kB writeback:0kB mapped:4612kB shmem:31268kB slab_reclaimable:23080kB slab_unreclaimable:452968kB kernel_stack:944kB pagetables:1084kB unstable:0kB bounce:0kB free_pcp:1768kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:48:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:48:01 oak-gw06 kernel: Node 0 Normal free:2703344kB min:323104kB low:403880kB high:484656kB active_anon:71160kB inactive_anon:168796kB active_file:1825796kB inactive_file:4944896kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:7548kB writeback:5788kB mapped:35176kB shmem:149044kB slab_reclaimable:135348kB slab_unreclaimable:3195340kB kernel_stack:4752kB pagetables:5640kB unstable:0kB bounce:0kB free_pcp:1868kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:48:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:48:01 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:48:01 oak-gw06 kernel: Node 0 DMA32: 7521*4kB (UEM) 11296*8kB (UEM) 12315*16kB (UEM) 9012*32kB (UEM) 1181*64kB (UM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 682740kB Aug 14 20:48:01 oak-gw06 kernel: Node 0 Normal: 57499*4kB (UE) 75044*8kB (UEM) 46005*16kB (UEM) 29332*32kB (UEM) 2940*64kB (UM) 50*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2699612kB Aug 14 20:48:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:48:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:48:01 oak-gw06 kernel: 2089421 total pagecache pages Aug 14 20:48:01 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:48:01 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:48:01 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:48:01 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:48:01 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:48:01 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:48:01 oak-gw06 kernel: 127313 pages reserved Aug 14 20:48:01 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 20:48:01 oak-gw06 kernel: CPU: 6 PID: 3905 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:48:01 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:48:01 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:48:01 oak-gw06 kernel: 00000000000080d0 0000000068f894c0 ffff880123193808 ffffffff8168662f Aug 14 20:48:01 oak-gw06 kernel: ffff880123193898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 20:48:01 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880123193868 0000000068f894c0 Aug 14 20:48:01 oak-gw06 kernel: Call Trace: Aug 14 20:48:01 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:48:01 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:48:01 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 20:48:01 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:48:01 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:48:01 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:48:01 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:48:01 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 20:48:01 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 20:48:01 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:48:01 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:48:01 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:48:01 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:48:01 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:48:01 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:48:01 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:48:01 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:48:01 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:48:01 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:48:01 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:48:01 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:48:01 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:48:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:48:01 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:48:01 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:48:01 oak-gw06 kernel: Mem-Info: Aug 14 20:48:01 oak-gw06 kernel: active_anon:20438 inactive_anon:51096 isolated_anon:0#012 active_file:568970 inactive_file:1534032 isolated_file:0#012 unevictable:0 dirty:2021 writeback:1522 unstable:0#012 slab_reclaimable:39607 slab_unreclaimable:912081#012 mapped:9947 shmem:45078 pagetables:1681 bounce:0#012 free:846226 free_pcp:490 free_cma:0 Aug 14 20:48:01 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:48:01 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:48:01 oak-gw06 kernel: Node 0 DMA32 free:683992kB min:69724kB low:87152kB high:104584kB active_anon:10592kB inactive_anon:35588kB active_file:450084kB inactive_file:1175632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1312kB writeback:0kB mapped:4612kB shmem:31268kB slab_reclaimable:23080kB slab_unreclaimable:452968kB kernel_stack:944kB pagetables:1084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:48:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:48:01 oak-gw06 kernel: Node 0 Normal free:2677204kB min:323104kB low:403880kB high:484656kB active_anon:71420kB inactive_anon:168796kB active_file:1825796kB inactive_file:4965956kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:6772kB writeback:4236kB mapped:35176kB shmem:149044kB slab_reclaimable:135348kB slab_unreclaimable:3195340kB kernel_stack:4752kB pagetables:5640kB unstable:0kB bounce:0kB free_pcp:2320kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:48:01 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:48:01 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:48:01 oak-gw06 kernel: Node 0 DMA32: 8076*4kB (UEM) 11333*8kB (UEM) 12519*16kB (UEM) 9012*32kB (UEM) 1181*64kB (UM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 688520kB Aug 14 20:48:01 oak-gw06 kernel: Node 0 Normal: 57119*4kB (UEM) 72181*8kB (UEM) 45869*16kB (UEM) 29344*32kB (UEM) 2941*64kB (UM) 50*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2673460kB Aug 14 20:48:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:48:01 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:48:01 oak-gw06 kernel: 2095144 total pagecache pages Aug 14 20:48:01 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:48:01 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:48:01 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:48:01 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:48:01 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:48:01 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:48:01 oak-gw06 kernel: 127313 pages reserved Aug 14 20:53:03 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 20:53:03 oak-gw06 kernel: CPU: 6 PID: 3893 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:53:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:53:03 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:53:03 oak-gw06 kernel: 00000000000080d0 00000000edecd00c ffff88019d533858 ffffffff8168662f Aug 14 20:53:03 oak-gw06 kernel: ffff88019d5338e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 14 20:53:03 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff88019d5338e8 00000000edecd00c Aug 14 20:53:03 oak-gw06 kernel: Call Trace: Aug 14 20:53:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:53:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:53:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:53:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:53:03 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 20:53:03 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 20:53:03 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:53:03 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:53:03 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:53:03 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:53:03 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:53:03 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:53:03 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:53:03 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:53:03 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:53:03 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:53:03 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:53:03 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:53:03 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:53:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:53:03 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:53:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:53:03 oak-gw06 kernel: Mem-Info: Aug 14 20:53:03 oak-gw06 kernel: active_anon:24820 inactive_anon:51096 isolated_anon:0#012 active_file:788360 inactive_file:1195850 isolated_file:0#012 unevictable:0 dirty:2650 writeback:1188 unstable:0#012 slab_reclaimable:36529 slab_unreclaimable:878900#012 mapped:9968 shmem:45078 pagetables:1692 bounce:0#012 free:980735 free_pcp:2017 free_cma:0 Aug 14 20:53:03 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:53:03 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:53:03 oak-gw06 kernel: Node 0 DMA32 free:920656kB min:69724kB low:87152kB high:104584kB active_anon:10684kB inactive_anon:35588kB active_file:591900kB inactive_file:820424kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2944kB writeback:2268kB mapped:4600kB shmem:31268kB slab_reclaimable:21288kB slab_unreclaimable:417032kB kernel_stack:944kB pagetables:1088kB unstable:0kB bounce:0kB free_pcp:3868kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:53:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:53:03 oak-gw06 kernel: Node 0 Normal free:2979228kB min:323104kB low:403880kB high:484656kB active_anon:88596kB inactive_anon:168796kB active_file:2561540kB inactive_file:3968584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:6980kB writeback:4916kB mapped:35272kB shmem:149044kB slab_reclaimable:124828kB slab_unreclaimable:3098968kB kernel_stack:4768kB pagetables:5680kB unstable:0kB bounce:0kB free_pcp:4592kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:53:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:53:03 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:53:03 oak-gw06 kernel: Node 0 DMA32: 7299*4kB (UEM) 8404*8kB (UEM) 19217*16kB (UEM) 10948*32kB (UEM) 2053*64kB (UM) 219*128kB (UM) 2*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 914172kB Aug 14 20:53:03 oak-gw06 kernel: Node 0 Normal: 48277*4kB (UE) 42501*8kB (UEM) 65601*16kB (UEM) 35620*32kB (UEM) 3764*64kB (UEM) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2970252kB Aug 14 20:53:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:53:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:53:03 oak-gw06 kernel: 1849861 total pagecache pages Aug 14 20:53:03 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:53:03 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:53:03 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:53:03 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:53:03 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:53:03 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:53:03 oak-gw06 kernel: 127313 pages reserved Aug 14 20:53:03 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 20:53:03 oak-gw06 kernel: CPU: 6 PID: 3893 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:53:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:53:03 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:53:03 oak-gw06 kernel: 00000000000080d0 00000000edecd00c ffff88019d533808 ffffffff8168662f Aug 14 20:53:03 oak-gw06 kernel: ffff88019d533898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 20:53:03 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019d533868 00000000edecd00c Aug 14 20:53:03 oak-gw06 kernel: Call Trace: Aug 14 20:53:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:53:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:53:03 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 20:53:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:53:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:53:03 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:53:03 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:53:03 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 20:53:03 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 20:53:03 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:53:03 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:53:03 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:53:03 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:53:03 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:53:03 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:53:03 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:53:03 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:53:03 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:53:03 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:53:03 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:53:03 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:53:03 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:53:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:53:03 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:53:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:53:03 oak-gw06 kernel: Mem-Info: Aug 14 20:53:03 oak-gw06 kernel: active_anon:24820 inactive_anon:51096 isolated_anon:0#012 active_file:788360 inactive_file:1203625 isolated_file:0#012 unevictable:0 dirty:2917 writeback:668 unstable:0#012 slab_reclaimable:36529 slab_unreclaimable:879850#012 mapped:9968 shmem:45078 pagetables:1692 bounce:0#012 free:969546 free_pcp:777 free_cma:0 Aug 14 20:53:03 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:53:03 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:53:03 oak-gw06 kernel: Node 0 DMA32 free:923404kB min:69724kB low:87152kB high:104584kB active_anon:10684kB inactive_anon:35588kB active_file:591900kB inactive_file:822776kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2360kB writeback:472kB mapped:4600kB shmem:31268kB slab_reclaimable:21288kB slab_unreclaimable:417448kB kernel_stack:944kB pagetables:1088kB unstable:0kB bounce:0kB free_pcp:1432kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:53:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:53:03 oak-gw06 kernel: Node 0 Normal free:2926600kB min:323104kB low:403880kB high:484656kB active_anon:88596kB inactive_anon:168796kB active_file:2561540kB inactive_file:4001084kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8920kB writeback:6080kB mapped:35272kB shmem:149044kB slab_reclaimable:124828kB slab_unreclaimable:3103280kB kernel_stack:4768kB pagetables:5680kB unstable:0kB bounce:0kB free_pcp:3016kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:53:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:53:03 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:53:03 oak-gw06 kernel: Node 0 DMA32: 8648*4kB (UE) 8400*8kB (UE) 19562*16kB (UEM) 10948*32kB (UEM) 2053*64kB (UM) 219*128kB (UM) 2*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 925056kB Aug 14 20:53:03 oak-gw06 kernel: Node 0 Normal: 47320*4kB (UEM) 42495*8kB (UEM) 62751*16kB (UEM) 35620*32kB (UEM) 3764*64kB (UEM) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2920776kB Aug 14 20:53:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:53:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:53:03 oak-gw06 kernel: 1858082 total pagecache pages Aug 14 20:53:03 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:53:03 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:53:03 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:53:03 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:53:03 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:53:03 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:53:03 oak-gw06 kernel: 127313 pages reserved Aug 14 20:58:02 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 20:58:02 oak-gw06 kernel: CPU: 6 PID: 3919 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:58:02 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:58:02 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:58:02 oak-gw06 kernel: 00000000000080d0 00000000c6a86a50 ffff88040cbeb858 ffffffff8168662f Aug 14 20:58:02 oak-gw06 kernel: ffff88040cbeb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 20:58:02 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88040cbeb8b8 00000000c6a86a50 Aug 14 20:58:02 oak-gw06 kernel: Call Trace: Aug 14 20:58:02 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:58:02 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:58:02 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:58:02 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:58:02 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 20:58:02 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 20:58:02 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:58:02 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:58:02 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:58:02 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:58:02 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:58:02 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:58:02 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:58:02 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:58:02 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:58:02 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:58:02 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:58:02 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:58:02 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:58:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:58:02 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:58:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:58:02 oak-gw06 kernel: Mem-Info: Aug 14 20:58:02 oak-gw06 kernel: active_anon:16044 inactive_anon:51096 isolated_anon:0#012 active_file:530238 inactive_file:1971167 isolated_file:0#012 unevictable:0 dirty:2554 writeback:3201 unstable:0#012 slab_reclaimable:36484 slab_unreclaimable:904348#012 mapped:9978 shmem:45078 pagetables:1393 bounce:0#012 free:463534 free_pcp:640 free_cma:0 Aug 14 20:58:02 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:58:02 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:58:02 oak-gw06 kernel: Node 0 DMA32 free:480516kB min:69724kB low:87152kB high:104584kB active_anon:10208kB inactive_anon:35588kB active_file:405272kB inactive_file:1451596kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2964kB writeback:836kB mapped:4596kB shmem:31268kB slab_reclaimable:21272kB slab_unreclaimable:429932kB kernel_stack:944kB pagetables:1076kB unstable:0kB bounce:0kB free_pcp:1168kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:58:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:58:02 oak-gw06 kernel: Node 0 Normal free:1348960kB min:323104kB low:403880kB high:484656kB active_anon:53236kB inactive_anon:168796kB active_file:1715908kB inactive_file:6438176kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:7468kB writeback:8340kB mapped:35316kB shmem:149044kB slab_reclaimable:124664kB slab_unreclaimable:3187140kB kernel_stack:4752kB pagetables:4828kB unstable:0kB bounce:0kB free_pcp:2032kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:58:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:58:02 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:58:02 oak-gw06 kernel: Node 0 DMA32: 9175*4kB (UEM) 7580*8kB (UEM) 1863*16kB (UEM) 4277*32kB (UEM) 3045*64kB (UM) 180*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 481932kB Aug 14 20:58:02 oak-gw06 kernel: Node 0 Normal: 50101*4kB (UE) 40314*8kB (UEM) 11646*16kB (UEM) 15608*32kB (UEM) 2145*64kB (UM) 15*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1347908kB Aug 14 20:58:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:58:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:58:02 oak-gw06 kernel: 2074330 total pagecache pages Aug 14 20:58:02 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:58:02 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:58:02 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:58:02 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:58:02 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:58:02 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:58:02 oak-gw06 kernel: 127313 pages reserved Aug 14 20:58:02 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 20:58:02 oak-gw06 kernel: CPU: 6 PID: 3919 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 20:58:02 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 20:58:02 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 20:58:02 oak-gw06 kernel: 00000000000080d0 00000000c6a86a50 ffff88040cbeb808 ffffffff8168662f Aug 14 20:58:02 oak-gw06 kernel: ffff88040cbeb898 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 14 20:58:02 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff88040cbeb898 00000000c6a86a50 Aug 14 20:58:02 oak-gw06 kernel: Call Trace: Aug 14 20:58:02 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 20:58:02 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 20:58:02 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 20:58:02 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 20:58:02 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 20:58:02 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 20:58:02 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 20:58:02 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 20:58:02 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 20:58:02 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 20:58:02 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 20:58:02 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 20:58:02 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 20:58:02 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 20:58:02 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 20:58:02 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 20:58:02 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 20:58:02 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 20:58:02 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 20:58:02 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 20:58:02 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 20:58:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:58:02 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 20:58:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 20:58:02 oak-gw06 kernel: Mem-Info: Aug 14 20:58:02 oak-gw06 kernel: active_anon:15818 inactive_anon:51096 isolated_anon:0#012 active_file:530232 inactive_file:1978058 isolated_file:0#012 unevictable:0 dirty:2579 writeback:1660 unstable:0#012 slab_reclaimable:36484 slab_unreclaimable:904200#012 mapped:9978 shmem:45078 pagetables:1475 bounce:0#012 free:457413 free_pcp:431 free_cma:0 Aug 14 20:58:02 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 20:58:02 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 20:58:02 oak-gw06 kernel: Node 0 DMA32 free:485852kB min:69724kB low:87152kB high:104584kB active_anon:10036kB inactive_anon:35588kB active_file:405272kB inactive_file:1451384kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2932kB writeback:0kB mapped:4596kB shmem:31268kB slab_reclaimable:21272kB slab_unreclaimable:429708kB kernel_stack:944kB pagetables:1072kB unstable:0kB bounce:0kB free_pcp:624kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:58:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 20:58:02 oak-gw06 kernel: Node 0 Normal free:1326912kB min:323104kB low:403880kB high:484656kB active_anon:53756kB inactive_anon:168796kB active_file:1715656kB inactive_file:6461108kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:7384kB writeback:6640kB mapped:35316kB shmem:149044kB slab_reclaimable:124664kB slab_unreclaimable:3187076kB kernel_stack:4752kB pagetables:4828kB unstable:0kB bounce:0kB free_pcp:1416kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 20:58:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 20:58:02 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 20:58:02 oak-gw06 kernel: Node 0 DMA32: 9535*4kB (UEM) 7582*8kB (UEM) 2063*16kB (UEM) 4278*32kB (UEM) 3045*64kB (UM) 180*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 486620kB Aug 14 20:58:02 oak-gw06 kernel: Node 0 Normal: 49840*4kB (UE) 38882*8kB (UEM) 11049*16kB (UEM) 15589*32kB (UEM) 2137*64kB (UM) 15*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1324736kB Aug 14 20:58:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 20:58:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 20:58:02 oak-gw06 kernel: 2079409 total pagecache pages Aug 14 20:58:02 oak-gw06 kernel: 16 pages in swap cache Aug 14 20:58:02 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 20:58:02 oak-gw06 kernel: Free swap = 4194036kB Aug 14 20:58:02 oak-gw06 kernel: Total swap = 4194300kB Aug 14 20:58:02 oak-gw06 kernel: 4194203 pages RAM Aug 14 20:58:02 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 20:58:02 oak-gw06 kernel: 127313 pages reserved Aug 14 21:03:02 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 21:03:02 oak-gw06 kernel: CPU: 6 PID: 3919 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:03:02 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:03:02 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:03:02 oak-gw06 kernel: 00000000000080d0 00000000c6a86a50 ffff88040cbeb858 ffffffff8168662f Aug 14 21:03:02 oak-gw06 kernel: ffff88040cbeb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 21:03:02 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88040cbeb8b8 00000000c6a86a50 Aug 14 21:03:02 oak-gw06 kernel: Call Trace: Aug 14 21:03:02 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:03:02 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:03:02 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:03:02 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:03:02 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 21:03:02 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 21:03:02 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:03:02 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:03:02 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:03:02 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:03:02 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:03:02 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:03:02 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:03:02 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:03:02 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:03:02 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:03:02 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:03:02 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:03:02 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:03:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:03:02 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:03:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:03:02 oak-gw06 kernel: Mem-Info: Aug 14 21:03:02 oak-gw06 kernel: active_anon:24827 inactive_anon:51096 isolated_anon:0#012 active_file:791904 inactive_file:1549427 isolated_file:0#012 unevictable:0 dirty:2918 writeback:1883 unstable:0#012 slab_reclaimable:36425 slab_unreclaimable:893784#012 mapped:9992 shmem:45078 pagetables:1688 bounce:0#012 free:624063 free_pcp:896 free_cma:0 Aug 14 21:03:02 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:03:02 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:03:02 oak-gw06 kernel: Node 0 DMA32 free:654064kB min:69724kB low:87152kB high:104584kB active_anon:11916kB inactive_anon:35588kB active_file:583372kB inactive_file:1080416kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2076kB writeback:1404kB mapped:4604kB shmem:31268kB slab_reclaimable:21268kB slab_unreclaimable:443736kB kernel_stack:944kB pagetables:1080kB unstable:0kB bounce:0kB free_pcp:1188kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:03:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:03:02 oak-gw06 kernel: Node 0 Normal free:1818248kB min:323104kB low:403880kB high:484656kB active_anon:87392kB inactive_anon:168796kB active_file:2584244kB inactive_file:5124832kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:7656kB writeback:11948kB mapped:35364kB shmem:149044kB slab_reclaimable:124432kB slab_unreclaimable:3131384kB kernel_stack:4768kB pagetables:5672kB unstable:0kB bounce:0kB free_pcp:3092kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:03:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:03:02 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:03:02 oak-gw06 kernel: Node 0 DMA32: 9602*4kB (UEM) 8382*8kB (UEM) 1660*16kB (UEM) 8840*32kB (UEM) 3211*64kB (UM) 278*128kB (UM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 656504kB Aug 14 21:03:02 oak-gw06 kernel: Node 0 Normal: 55362*4kB (UEM) 42019*8kB (UE) 12003*16kB (UEM) 23448*32kB (UEM) 4724*64kB (UEM) 79*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1812432kB Aug 14 21:03:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:03:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:03:02 oak-gw06 kernel: 2035670 total pagecache pages Aug 14 21:03:02 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:03:02 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:03:02 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:03:02 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:03:02 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:03:02 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:03:02 oak-gw06 kernel: 127313 pages reserved Aug 14 21:03:02 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 21:03:02 oak-gw06 kernel: CPU: 7 PID: 3919 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:03:02 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:03:02 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:03:02 oak-gw06 kernel: 00000000000080d0 00000000c6a86a50 ffff88040cbeb808 ffffffff8168662f Aug 14 21:03:02 oak-gw06 kernel: ffff88040cbeb898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 21:03:02 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88040cbeb868 00000000c6a86a50 Aug 14 21:03:02 oak-gw06 kernel: Call Trace: Aug 14 21:03:02 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:03:02 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:03:02 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:03:02 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:03:02 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 21:03:02 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 21:03:02 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 21:03:02 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 21:03:02 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:03:02 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:03:02 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:03:02 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:03:02 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:03:02 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:03:02 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:03:02 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:03:02 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:03:02 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:03:02 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:03:02 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:03:02 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:03:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:03:02 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:03:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:03:02 oak-gw06 kernel: Mem-Info: Aug 14 21:03:02 oak-gw06 kernel: active_anon:24827 inactive_anon:51096 isolated_anon:0#012 active_file:791839 inactive_file:1556586 isolated_file:0#012 unevictable:0 dirty:2627 writeback:3242 unstable:0#012 slab_reclaimable:36425 slab_unreclaimable:893784#012 mapped:9993 shmem:45078 pagetables:1688 bounce:0#012 free:617101 free_pcp:442 free_cma:0 Aug 14 21:03:02 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:03:02 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:03:02 oak-gw06 kernel: Node 0 DMA32 free:661884kB min:69724kB low:87152kB high:104584kB active_anon:11916kB inactive_anon:35588kB active_file:583372kB inactive_file:1080408kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2076kB writeback:1404kB mapped:4604kB shmem:31268kB slab_reclaimable:21268kB slab_unreclaimable:443736kB kernel_stack:944kB pagetables:1080kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:03:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:03:02 oak-gw06 kernel: Node 0 Normal free:1777208kB min:323104kB low:403880kB high:484656kB active_anon:87392kB inactive_anon:168796kB active_file:2583984kB inactive_file:5161016kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:13088kB writeback:11564kB mapped:35368kB shmem:149044kB slab_reclaimable:124432kB slab_unreclaimable:3131384kB kernel_stack:4768kB pagetables:5672kB unstable:0kB bounce:0kB free_pcp:2332kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:03:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:03:02 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:03:02 oak-gw06 kernel: Node 0 DMA32: 9369*4kB (UEM) 8315*8kB (UEM) 1673*16kB (UEM) 8842*32kB (UEM) 3211*64kB (UM) 278*128kB (UM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 655308kB Aug 14 21:03:02 oak-gw06 kernel: Node 0 Normal: 56203*4kB (UEM) 42027*8kB (UEM) 9822*16kB (UEM) 23454*32kB (UEM) 4724*64kB (UEM) 79*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1781156kB Aug 14 21:03:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:03:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:03:02 oak-gw06 kernel: 2045945 total pagecache pages Aug 14 21:03:02 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:03:02 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:03:02 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:03:02 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:03:02 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:03:02 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:03:02 oak-gw06 kernel: 127313 pages reserved Aug 14 21:08:02 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 21:08:02 oak-gw06 kernel: CPU: 3 PID: 3893 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:08:02 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:08:02 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:08:02 oak-gw06 kernel: 00000000000080d0 00000000edecd00c ffff88019d533858 ffffffff8168662f Aug 14 21:08:02 oak-gw06 kernel: ffff88019d5338e8 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Aug 14 21:08:02 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 00000000edecd00c Aug 14 21:08:02 oak-gw06 kernel: Call Trace: Aug 14 21:08:02 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:08:02 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:08:02 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Aug 14 21:08:02 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:08:02 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:08:02 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 21:08:02 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 21:08:02 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:08:02 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:08:02 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:08:02 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:08:02 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:08:02 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:08:02 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:08:02 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:08:02 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:08:02 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:08:02 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:08:02 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:08:02 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:08:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:08:02 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:08:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:08:02 oak-gw06 kernel: Mem-Info: Aug 14 21:08:02 oak-gw06 kernel: active_anon:24827 inactive_anon:51096 isolated_anon:0#012 active_file:843400 inactive_file:1406860 isolated_file:0#012 unevictable:0 dirty:2939 writeback:1565 unstable:0#012 slab_reclaimable:36398 slab_unreclaimable:893093#012 mapped:10007 shmem:45078 pagetables:1688 bounce:0#012 free:706655 free_pcp:1212 free_cma:0 Aug 14 21:08:02 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:08:02 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:08:02 oak-gw06 kernel: Node 0 DMA32 free:774580kB min:69724kB low:87152kB high:104584kB active_anon:11924kB inactive_anon:35588kB active_file:644716kB inactive_file:917708kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3272kB writeback:148kB mapped:4612kB shmem:31268kB slab_reclaimable:21268kB slab_unreclaimable:426732kB kernel_stack:912kB pagetables:1088kB unstable:0kB bounce:0kB free_pcp:1472kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:08:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:08:02 oak-gw06 kernel: Node 0 Normal free:2040232kB min:323104kB low:403880kB high:484656kB active_anon:87384kB inactive_anon:168796kB active_file:2728884kB inactive_file:4712852kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8484kB writeback:6500kB mapped:35416kB shmem:149044kB slab_reclaimable:124324kB slab_unreclaimable:3145624kB kernel_stack:4784kB pagetables:5664kB unstable:0kB bounce:0kB free_pcp:3948kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:08:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:08:02 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:08:02 oak-gw06 kernel: Node 0 DMA32: 10464*4kB (UEM) 8432*8kB (UEM) 7842*16kB (UEM) 8636*32kB (UEM) 3572*64kB (UM) 287*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 777248kB Aug 14 21:08:02 oak-gw06 kernel: Node 0 Normal: 50937*4kB (UEM) 43135*8kB (UEM) 27761*16kB (UEM) 25309*32kB (UEM) 3621*64kB (UM) 75*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2044236kB Aug 14 21:08:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:08:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:08:02 oak-gw06 kernel: 2080225 total pagecache pages Aug 14 21:08:02 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:08:02 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:08:02 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:08:02 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:08:02 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:08:02 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:08:02 oak-gw06 kernel: 127313 pages reserved Aug 14 21:08:02 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 21:08:02 oak-gw06 kernel: CPU: 3 PID: 3893 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:08:02 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:08:02 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:08:02 oak-gw06 kernel: 00000000000080d0 00000000edecd00c ffff88019d533808 ffffffff8168662f Aug 14 21:08:02 oak-gw06 kernel: ffff88019d533898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 21:08:02 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019d533868 00000000edecd00c Aug 14 21:08:02 oak-gw06 kernel: Call Trace: Aug 14 21:08:02 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:08:02 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:08:02 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:08:02 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:08:02 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 21:08:02 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 21:08:02 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 21:08:02 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 21:08:02 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:08:02 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:08:02 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:08:02 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:08:02 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:08:02 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:08:02 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:08:02 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:08:02 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:08:02 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:08:02 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:08:02 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:08:02 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:08:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:08:02 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:08:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:08:02 oak-gw06 kernel: Mem-Info: Aug 14 21:08:02 oak-gw06 kernel: active_anon:24827 inactive_anon:51096 isolated_anon:0#012 active_file:843400 inactive_file:1406863 isolated_file:0#012 unevictable:0 dirty:2745 writeback:886 unstable:0#012 slab_reclaimable:36398 slab_unreclaimable:892871#012 mapped:10007 shmem:45078 pagetables:1688 bounce:0#012 free:714415 free_pcp:1356 free_cma:0 Aug 14 21:08:02 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:08:02 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:08:02 oak-gw06 kernel: Node 0 DMA32 free:786096kB min:69724kB low:87152kB high:104584kB active_anon:11924kB inactive_anon:35588kB active_file:644716kB inactive_file:910240kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3272kB writeback:148kB mapped:4612kB shmem:31268kB slab_reclaimable:21268kB slab_unreclaimable:426380kB kernel_stack:912kB pagetables:1088kB unstable:0kB bounce:0kB free_pcp:2616kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:08:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:08:02 oak-gw06 kernel: Node 0 Normal free:2062836kB min:323104kB low:403880kB high:484656kB active_anon:87384kB inactive_anon:168796kB active_file:2728884kB inactive_file:4717532kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:7708kB writeback:2620kB mapped:35416kB shmem:149044kB slab_reclaimable:124324kB slab_unreclaimable:3144816kB kernel_stack:4784kB pagetables:5664kB unstable:0kB bounce:0kB free_pcp:2784kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:08:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:08:02 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:08:02 oak-gw06 kernel: Node 0 DMA32: 11349*4kB (UEM) 8626*8kB (UEM) 8162*16kB (UEM) 8646*32kB (UEM) 3573*64kB (UM) 287*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 787844kB Aug 14 21:08:02 oak-gw06 kernel: Node 0 Normal: 53998*4kB (UEM) 43075*8kB (UEM) 28264*16kB (UEM) 25329*32kB (UEM) 3623*64kB (UM) 75*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2064816kB Aug 14 21:08:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:08:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:08:02 oak-gw06 kernel: 2073747 total pagecache pages Aug 14 21:08:02 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:08:02 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:08:02 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:08:02 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:08:02 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:08:02 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:08:02 oak-gw06 kernel: 127313 pages reserved Aug 14 21:13:02 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 21:13:02 oak-gw06 kernel: CPU: 6 PID: 3893 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:13:02 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:13:02 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:13:02 oak-gw06 kernel: 00000000000080d0 00000000edecd00c ffff88019d533858 ffffffff8168662f Aug 14 21:13:02 oak-gw06 kernel: ffff88019d5338e8 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Aug 14 21:13:02 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 00000000edecd00c Aug 14 21:13:02 oak-gw06 kernel: Call Trace: Aug 14 21:13:02 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:13:02 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:13:02 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Aug 14 21:13:02 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:13:02 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:13:02 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 21:13:02 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 21:13:02 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:13:02 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:13:02 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:13:02 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:13:02 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:13:02 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:13:02 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:13:02 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:13:02 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:13:02 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:13:02 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:13:02 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:13:02 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:13:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:13:02 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:13:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:13:02 oak-gw06 kernel: Mem-Info: Aug 14 21:13:02 oak-gw06 kernel: active_anon:20222 inactive_anon:51096 isolated_anon:0#012 active_file:590699 inactive_file:1909406 isolated_file:0#012 unevictable:0 dirty:2573 writeback:2406 unstable:0#012 slab_reclaimable:36342 slab_unreclaimable:890659#012 mapped:10018 shmem:45078 pagetables:1660 bounce:0#012 free:472917 free_pcp:429 free_cma:0 Aug 14 21:13:02 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:13:02 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:13:02 oak-gw06 kernel: Node 0 DMA32 free:649452kB min:69724kB low:87152kB high:104584kB active_anon:15268kB inactive_anon:35588kB active_file:439120kB inactive_file:1255784kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:640kB writeback:0kB mapped:4616kB shmem:31268kB slab_reclaimable:21236kB slab_unreclaimable:418500kB kernel_stack:976kB pagetables:1868kB unstable:0kB bounce:0kB free_pcp:408kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:13:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:13:02 oak-gw06 kernel: Node 0 Normal free:1206156kB min:323104kB low:403880kB high:484656kB active_anon:66400kB inactive_anon:168796kB active_file:1923676kB inactive_file:6395880kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10816kB writeback:9432kB mapped:35456kB shmem:149044kB slab_reclaimable:124132kB slab_unreclaimable:3145752kB kernel_stack:4704kB pagetables:4772kB unstable:0kB bounce:0kB free_pcp:1344kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:13:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:13:02 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:13:02 oak-gw06 kernel: Node 0 DMA32: 9653*4kB (UEM) 8281*8kB (UEM) 2948*16kB (UEM) 7149*32kB (UEM) 3744*64kB (UM) 243*128kB (UM) 6*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 653052kB Aug 14 21:13:02 oak-gw06 kernel: Node 0 Normal: 41680*4kB (UEM) 30831*8kB (UE) 17861*16kB (UEM) 13537*32kB (UEM) 1090*64kB (UM) 7*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1202984kB Aug 14 21:13:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:13:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:13:02 oak-gw06 kernel: 2081376 total pagecache pages Aug 14 21:13:02 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:13:02 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:13:02 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:13:02 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:13:02 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:13:02 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:13:02 oak-gw06 kernel: 127313 pages reserved Aug 14 21:13:02 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 21:13:02 oak-gw06 kernel: CPU: 3 PID: 3893 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:13:02 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:13:02 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:13:02 oak-gw06 kernel: 00000000000080d0 00000000edecd00c ffff88019d533808 ffffffff8168662f Aug 14 21:13:02 oak-gw06 kernel: ffff88019d533898 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Aug 14 21:13:02 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 00000000edecd00c Aug 14 21:13:02 oak-gw06 kernel: Call Trace: Aug 14 21:13:02 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:13:02 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:13:02 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Aug 14 21:13:02 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:13:02 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:13:02 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 21:13:02 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 21:13:02 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 21:13:02 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 21:13:02 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:13:02 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:13:02 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:13:02 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:13:02 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:13:02 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:13:02 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:13:02 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:13:02 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:13:02 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:13:02 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:13:02 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:13:02 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:13:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:13:02 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:13:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:13:02 oak-gw06 kernel: Mem-Info: Aug 14 21:13:02 oak-gw06 kernel: active_anon:20287 inactive_anon:51096 isolated_anon:0#012 active_file:590829 inactive_file:1917076 isolated_file:0#012 unevictable:0 dirty:2379 writeback:4928 unstable:0#012 slab_reclaimable:36342 slab_unreclaimable:891475#012 mapped:10018 shmem:45078 pagetables:1660 bounce:0#012 free:462071 free_pcp:948 free_cma:0 Aug 14 21:13:02 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:13:02 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:13:02 oak-gw06 kernel: Node 0 DMA32 free:653024kB min:69724kB low:87152kB high:104584kB active_anon:15268kB inactive_anon:35588kB active_file:439120kB inactive_file:1255784kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:640kB writeback:0kB mapped:4616kB shmem:31268kB slab_reclaimable:21236kB slab_unreclaimable:418500kB kernel_stack:976kB pagetables:1868kB unstable:0kB bounce:0kB free_pcp:316kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:13:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:13:02 oak-gw06 kernel: Node 0 Normal free:1166344kB min:323104kB low:403880kB high:484656kB active_anon:67440kB inactive_anon:168796kB active_file:1924196kB inactive_file:6420060kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:11980kB writeback:20296kB mapped:35456kB shmem:149044kB slab_reclaimable:124132kB slab_unreclaimable:3148200kB kernel_stack:4704kB pagetables:4772kB unstable:0kB bounce:0kB free_pcp:4048kB local_pcp:76kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:13:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:13:02 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:13:02 oak-gw06 kernel: Node 0 DMA32: 9716*4kB (UEM) 8273*8kB (UE) 3103*16kB (UEM) 7150*32kB (UEM) 3744*64kB (UM) 243*128kB (UM) 6*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 655752kB Aug 14 21:13:02 oak-gw06 kernel: Node 0 Normal: 39710*4kB (UE) 30831*8kB (UE) 15128*16kB (UEM) 13552*32kB (UEM) 1090*64kB (UM) 7*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1151856kB Aug 14 21:13:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:13:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:13:02 oak-gw06 kernel: 2086620 total pagecache pages Aug 14 21:13:02 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:13:02 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:13:02 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:13:02 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:13:02 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:13:02 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:13:02 oak-gw06 kernel: 127313 pages reserved Aug 14 21:18:02 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 21:18:02 oak-gw06 kernel: CPU: 7 PID: 3893 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:18:02 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:18:02 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:18:02 oak-gw06 kernel: 00000000000080d0 00000000edecd00c ffff88019d533858 ffffffff8168662f Aug 14 21:18:02 oak-gw06 kernel: ffff88019d5338e8 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Aug 14 21:18:02 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 00000000edecd00c Aug 14 21:18:02 oak-gw06 kernel: Call Trace: Aug 14 21:18:02 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:18:02 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:18:02 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Aug 14 21:18:02 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:18:02 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:18:02 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 21:18:02 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 21:18:02 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:18:02 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:18:02 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:18:02 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:18:02 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:18:02 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:18:02 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:18:02 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:18:02 oak-gw06 kernel: [] ? finish_task_switch+0x56/0x180 Aug 14 21:18:02 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:18:02 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:18:02 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:18:02 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:18:02 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:18:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:18:02 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:18:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:18:02 oak-gw06 kernel: Mem-Info: Aug 14 21:18:02 oak-gw06 kernel: active_anon:25789 inactive_anon:51096 isolated_anon:0#012 active_file:1260944 inactive_file:600088 isolated_file:0#012 unevictable:0 dirty:2249 writeback:256 unstable:0#012 slab_reclaimable:35759 slab_unreclaimable:857009#012 mapped:10029 shmem:45078 pagetables:1691 bounce:0#012 free:1077844 free_pcp:418 free_cma:0 Aug 14 21:18:02 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:18:02 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:18:02 oak-gw06 kernel: Node 0 DMA32 free:931728kB min:69724kB low:87152kB high:104584kB active_anon:18960kB inactive_anon:35588kB active_file:891564kB inactive_file:482924kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2172kB writeback:0kB mapped:4616kB shmem:31268kB slab_reclaimable:20820kB slab_unreclaimable:408004kB kernel_stack:992kB pagetables:1864kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:18:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:18:02 oak-gw06 kernel: Node 0 Normal free:3359592kB min:323104kB low:403880kB high:484656kB active_anon:84196kB inactive_anon:168796kB active_file:4162596kB inactive_file:1907044kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:6824kB writeback:248kB mapped:35500kB shmem:149044kB slab_reclaimable:122216kB slab_unreclaimable:3020016kB kernel_stack:4704kB pagetables:4900kB unstable:0kB bounce:0kB free_pcp:1732kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:18:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:18:02 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:18:02 oak-gw06 kernel: Node 0 DMA32: 4747*4kB (UEM) 8103*8kB (UEM) 8832*16kB (UEM) 12741*32kB (UEM) 4036*64kB (UM) 322*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 932612kB Aug 14 21:18:02 oak-gw06 kernel: Node 0 Normal: 27660*4kB (UEM) 41855*8kB (UE) 15008*16kB (UEM) 59279*32kB (UEM) 11092*64kB (UEM) 504*128kB (UM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3357448kB Aug 14 21:18:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:18:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:18:02 oak-gw06 kernel: 1906291 total pagecache pages Aug 14 21:18:02 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:18:02 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:18:02 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:18:02 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:18:02 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:18:02 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:18:02 oak-gw06 kernel: 127313 pages reserved Aug 14 21:18:02 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 21:18:02 oak-gw06 kernel: CPU: 7 PID: 3893 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:18:02 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:18:02 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:18:02 oak-gw06 kernel: 00000000000080d0 00000000edecd00c ffff88019d533808 ffffffff8168662f Aug 14 21:18:02 oak-gw06 kernel: ffff88019d533898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 21:18:02 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019d533868 00000000edecd00c Aug 14 21:18:02 oak-gw06 kernel: Call Trace: Aug 14 21:18:02 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:18:02 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:18:02 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 21:18:02 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:18:02 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:18:02 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 21:18:02 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 21:18:02 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 21:18:02 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 21:18:02 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:18:02 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:18:02 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:18:02 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:18:02 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:18:02 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:18:02 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:18:02 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:18:02 oak-gw06 kernel: [] ? finish_task_switch+0x56/0x180 Aug 14 21:18:02 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:18:02 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:18:02 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:18:02 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:18:02 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:18:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:18:02 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:18:02 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:18:02 oak-gw06 kernel: Mem-Info: Aug 14 21:18:02 oak-gw06 kernel: active_anon:25789 inactive_anon:51096 isolated_anon:0#012 active_file:1271568 inactive_file:592974 isolated_file:0#012 unevictable:0 dirty:2346 writeback:2972 unstable:0#012 slab_reclaimable:35759 slab_unreclaimable:857349#012 mapped:10029 shmem:45078 pagetables:1691 bounce:0#012 free:1074471 free_pcp:289 free_cma:0 Aug 14 21:18:02 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:18:02 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:18:02 oak-gw06 kernel: Node 0 DMA32 free:935816kB min:69724kB low:87152kB high:104584kB active_anon:18960kB inactive_anon:35588kB active_file:895596kB inactive_file:478892kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2172kB writeback:0kB mapped:4616kB shmem:31268kB slab_reclaimable:20820kB slab_unreclaimable:408004kB kernel_stack:992kB pagetables:1864kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:18:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:18:02 oak-gw06 kernel: Node 0 Normal free:3338200kB min:323104kB low:403880kB high:484656kB active_anon:84196kB inactive_anon:168796kB active_file:4196136kB inactive_file:1892224kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:7600kB writeback:11112kB mapped:35500kB shmem:149044kB slab_reclaimable:122216kB slab_unreclaimable:3022192kB kernel_stack:4704kB pagetables:4900kB unstable:0kB bounce:0kB free_pcp:1452kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:18:02 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:18:02 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:18:02 oak-gw06 kernel: Node 0 DMA32: 4869*4kB (UEM) 8103*8kB (UEM) 9048*16kB (UEM) 12749*32kB (UEM) 4036*64kB (UM) 322*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 936812kB Aug 14 21:18:02 oak-gw06 kernel: Node 0 Normal: 27724*4kB (UEM) 41856*8kB (UEM) 13732*16kB (UEM) 59159*32kB (UEM) 11092*64kB (UEM) 504*128kB (UM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3333456kB Aug 14 21:18:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:18:02 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:18:02 oak-gw06 kernel: 1911432 total pagecache pages Aug 14 21:18:02 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:18:02 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:18:02 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:18:02 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:18:02 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:18:02 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:18:02 oak-gw06 kernel: 127313 pages reserved Aug 14 21:23:03 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 21:23:03 oak-gw06 kernel: CPU: 6 PID: 3893 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:23:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:23:03 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:23:03 oak-gw06 kernel: 00000000000080d0 00000000edecd00c ffff88019d533858 ffffffff8168662f Aug 14 21:23:03 oak-gw06 kernel: ffff88019d5338e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 21:23:03 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019d5338b8 00000000edecd00c Aug 14 21:23:03 oak-gw06 kernel: Call Trace: Aug 14 21:23:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:23:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:23:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:23:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:23:03 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 21:23:03 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 21:23:03 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:23:03 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:23:03 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:23:03 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:23:03 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:23:03 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:23:03 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:23:03 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:23:03 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:23:03 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:23:03 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:23:03 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:23:03 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:23:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:23:03 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:23:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:23:03 oak-gw06 kernel: Mem-Info: Aug 14 21:23:03 oak-gw06 kernel: active_anon:19982 inactive_anon:51096 isolated_anon:0#012 active_file:193397 inactive_file:2104848 isolated_file:0#012 unevictable:0 dirty:3211 writeback:3137 unstable:0#012 slab_reclaimable:35693 slab_unreclaimable:865602#012 mapped:10031 shmem:45078 pagetables:1673 bounce:0#012 free:702161 free_pcp:543 free_cma:0 Aug 14 21:23:03 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:23:03 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:23:03 oak-gw06 kernel: Node 0 DMA32 free:734404kB min:69724kB low:87152kB high:104584kB active_anon:10036kB inactive_anon:35588kB active_file:159856kB inactive_file:1456840kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2388kB writeback:1616kB mapped:4628kB shmem:31268kB slab_reclaimable:20788kB slab_unreclaimable:415636kB kernel_stack:928kB pagetables:1080kB unstable:0kB bounce:0kB free_pcp:1208kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:23:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:23:03 oak-gw06 kernel: Node 0 Normal free:2055056kB min:323104kB low:403880kB high:484656kB active_anon:70152kB inactive_anon:168796kB active_file:613576kB inactive_file:6965472kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9988kB writeback:10756kB mapped:35504kB shmem:149044kB slab_reclaimable:121984kB slab_unreclaimable:3047364kB kernel_stack:4784kB pagetables:5612kB unstable:0kB bounce:0kB free_pcp:1588kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:23:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:23:03 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:23:03 oak-gw06 kernel: Node 0 DMA32: 9545*4kB (UEM) 8231*8kB (UEM) 6231*16kB (UEM) 9074*32kB (UEM) 2865*64kB (UM) 381*128kB (UM) 32*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 734412kB Aug 14 21:23:03 oak-gw06 kernel: Node 0 Normal: 43033*4kB (UEM) 32310*8kB (UE) 32261*16kB (UEM) 28106*32kB (UEM) 3044*64kB (UM) 77*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2050852kB Aug 14 21:23:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:23:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:23:03 oak-gw06 kernel: 2053598 total pagecache pages Aug 14 21:23:03 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:23:03 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:23:03 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:23:03 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:23:03 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:23:03 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:23:03 oak-gw06 kernel: 127313 pages reserved Aug 14 21:23:03 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 21:23:03 oak-gw06 kernel: CPU: 6 PID: 3893 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:23:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:23:03 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:23:03 oak-gw06 kernel: 00000000000080d0 00000000edecd00c ffff88019d533808 ffffffff8168662f Aug 14 21:23:03 oak-gw06 kernel: ffff88019d533898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 21:23:03 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019d533868 00000000edecd00c Aug 14 21:23:03 oak-gw06 kernel: Call Trace: Aug 14 21:23:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:23:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:23:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:23:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:23:03 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 21:23:03 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 21:23:03 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 21:23:03 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 21:23:03 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:23:03 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:23:03 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:23:03 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:23:03 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:23:03 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:23:03 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:23:03 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:23:03 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:23:03 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:23:03 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:23:03 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:23:03 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:23:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:23:03 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:23:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:23:03 oak-gw06 kernel: Mem-Info: Aug 14 21:23:03 oak-gw06 kernel: active_anon:19982 inactive_anon:51096 isolated_anon:0#012 active_file:193327 inactive_file:2110953 isolated_file:0#012 unevictable:0 dirty:2928 writeback:3363 unstable:0#012 slab_reclaimable:35693 slab_unreclaimable:866350#012 mapped:10033 shmem:45078 pagetables:1673 bounce:0#012 free:695131 free_pcp:623 free_cma:0 Aug 14 21:23:03 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:23:03 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:23:03 oak-gw06 kernel: Node 0 DMA32 free:740020kB min:69724kB low:87152kB high:104584kB active_anon:10036kB inactive_anon:35588kB active_file:159796kB inactive_file:1456972kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2264kB writeback:1536kB mapped:4628kB shmem:31268kB slab_reclaimable:20788kB slab_unreclaimable:415508kB kernel_stack:928kB pagetables:1080kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:23:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:23:03 oak-gw06 kernel: Node 0 Normal free:2016728kB min:323104kB low:403880kB high:484656kB active_anon:69892kB inactive_anon:168796kB active_file:614552kB inactive_file:6991000kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9836kB writeback:16184kB mapped:35504kB shmem:149044kB slab_reclaimable:121984kB slab_unreclaimable:3050420kB kernel_stack:4784kB pagetables:5612kB unstable:0kB bounce:0kB free_pcp:3028kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:23:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:23:03 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:23:03 oak-gw06 kernel: Node 0 DMA32: 9847*4kB (UEM) 8231*8kB (UEM) 6561*16kB (UEM) 9074*32kB (UEM) 2865*64kB (UM) 381*128kB (UM) 32*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 740900kB Aug 14 21:23:03 oak-gw06 kernel: Node 0 Normal: 42608*4kB (UEM) 32311*8kB (UE) 30052*16kB (UEM) 28107*32kB (UEM) 3045*64kB (UM) 77*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2013912kB Aug 14 21:23:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:23:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:23:03 oak-gw06 kernel: 2059926 total pagecache pages Aug 14 21:23:03 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:23:03 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:23:03 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:23:03 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:23:03 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:23:03 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:23:03 oak-gw06 kernel: 127313 pages reserved Aug 14 21:28:03 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 21:28:03 oak-gw06 kernel: CPU: 6 PID: 3893 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:28:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:28:03 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:28:03 oak-gw06 kernel: 00000000000080d0 00000000edecd00c ffff88019d533858 ffffffff8168662f Aug 14 21:28:03 oak-gw06 kernel: ffff88019d5338e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 21:28:03 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019d5338b8 00000000edecd00c Aug 14 21:28:03 oak-gw06 kernel: Call Trace: Aug 14 21:28:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:28:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:28:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:28:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:28:03 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 21:28:03 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 21:28:03 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:28:03 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:28:03 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:28:03 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:28:03 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:28:03 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:28:03 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:28:03 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:28:03 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:28:03 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:28:03 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:28:03 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:28:03 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:28:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:28:03 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:28:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:28:03 oak-gw06 kernel: Mem-Info: Aug 14 21:28:03 oak-gw06 kernel: active_anon:24588 inactive_anon:51096 isolated_anon:0#012 active_file:271406 inactive_file:2130432 isolated_file:0#012 unevictable:0 dirty:4989 writeback:3569 unstable:0#012 slab_reclaimable:35485 slab_unreclaimable:857220#012 mapped:10050 shmem:45078 pagetables:1687 bounce:0#012 free:602158 free_pcp:408 free_cma:0 Aug 14 21:28:03 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:28:03 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:28:03 oak-gw06 kernel: Node 0 DMA32 free:600048kB min:69724kB low:87152kB high:104584kB active_anon:11292kB inactive_anon:35588kB active_file:214952kB inactive_file:1527232kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2168kB writeback:896kB mapped:4620kB shmem:31268kB slab_reclaimable:20684kB slab_unreclaimable:424812kB kernel_stack:928kB pagetables:1104kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:28:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:28:03 oak-gw06 kernel: Node 0 Normal free:1779856kB min:323104kB low:403880kB high:484656kB active_anon:87060kB inactive_anon:168796kB active_file:874816kB inactive_file:7003872kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10028kB writeback:15416kB mapped:35580kB shmem:149044kB slab_reclaimable:121256kB slab_unreclaimable:3004052kB kernel_stack:4752kB pagetables:5644kB unstable:0kB bounce:0kB free_pcp:2044kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:28:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:28:03 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:28:03 oak-gw06 kernel: Node 0 DMA32: 9923*4kB (UEM) 11564*8kB (UEM) 1899*16kB (UEM) 6541*32kB (UEM) 3038*64kB (UM) 270*128kB (UM) 4*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 601916kB Aug 14 21:28:03 oak-gw06 kernel: Node 0 Normal: 52663*4kB (UE) 53207*8kB (UEM) 10243*16kB (UEM) 22099*32kB (UEM) 4087*64kB (UEM) 73*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1778276kB Aug 14 21:28:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:28:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:28:03 oak-gw06 kernel: 2083574 total pagecache pages Aug 14 21:28:03 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:28:03 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:28:03 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:28:03 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:28:03 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:28:03 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:28:03 oak-gw06 kernel: 127313 pages reserved Aug 14 21:28:03 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 21:28:03 oak-gw06 kernel: CPU: 6 PID: 3893 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:28:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:28:03 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:28:03 oak-gw06 kernel: 00000000000080d0 00000000edecd00c ffff88019d533808 ffffffff8168662f Aug 14 21:28:03 oak-gw06 kernel: ffff88019d533898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 21:28:03 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019d533868 00000000edecd00c Aug 14 21:28:03 oak-gw06 kernel: Call Trace: Aug 14 21:28:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:28:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:28:03 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 21:28:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:28:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:28:03 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 21:28:03 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 21:28:03 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 21:28:03 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 21:28:03 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:28:03 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:28:03 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:28:03 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:28:03 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:28:03 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:28:03 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:28:03 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:28:03 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:28:03 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:28:03 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:28:03 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:28:03 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:28:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:28:03 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:28:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:28:03 oak-gw06 kernel: Mem-Info: Aug 14 21:28:03 oak-gw06 kernel: active_anon:24588 inactive_anon:51096 isolated_anon:0#012 active_file:277598 inactive_file:2132365 isolated_file:0#012 unevictable:0 dirty:2855 writeback:2720 unstable:0#012 slab_reclaimable:35485 slab_unreclaimable:857356#012 mapped:10050 shmem:45078 pagetables:1687 bounce:0#012 free:593066 free_pcp:371 free_cma:0 Aug 14 21:28:03 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:28:03 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:28:03 oak-gw06 kernel: Node 0 DMA32 free:605148kB min:69724kB low:87152kB high:104584kB active_anon:11292kB inactive_anon:35588kB active_file:219992kB inactive_file:1520680kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2168kB writeback:896kB mapped:4620kB shmem:31268kB slab_reclaimable:20684kB slab_unreclaimable:424812kB kernel_stack:928kB pagetables:1104kB unstable:0kB bounce:0kB free_pcp:1300kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:28:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:28:03 oak-gw06 kernel: Node 0 Normal free:1739288kB min:323104kB low:403880kB high:484656kB active_anon:87060kB inactive_anon:168796kB active_file:896136kB inactive_file:7016352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8476kB writeback:14640kB mapped:35580kB shmem:149044kB slab_reclaimable:121256kB slab_unreclaimable:3005676kB kernel_stack:4752kB pagetables:5644kB unstable:0kB bounce:0kB free_pcp:1412kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:28:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:28:03 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:28:03 oak-gw06 kernel: Node 0 DMA32: 9926*4kB (UEM) 11698*8kB (UEM) 2145*16kB (UEM) 6544*32kB (UEM) 3038*64kB (UM) 270*128kB (UM) 4*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 607032kB Aug 14 21:28:03 oak-gw06 kernel: Node 0 Normal: 52717*4kB (UEM) 48921*8kB (UEM) 9787*16kB (UEM) 22127*32kB (UEM) 4087*64kB (UEM) 73*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1737804kB Aug 14 21:28:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:28:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:28:03 oak-gw06 kernel: 2088830 total pagecache pages Aug 14 21:28:03 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:28:03 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:28:03 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:28:03 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:28:03 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:28:03 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:28:03 oak-gw06 kernel: 127313 pages reserved Aug 14 21:33:03 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 21:33:03 oak-gw06 kernel: CPU: 6 PID: 3995 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:33:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:33:03 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:33:03 oak-gw06 kernel: 00000000000080d0 00000000b3e284da ffff8801df52b858 ffffffff8168662f Aug 14 21:33:03 oak-gw06 kernel: ffff8801df52b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 21:33:03 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801df52b8b8 00000000b3e284da Aug 14 21:33:03 oak-gw06 kernel: Call Trace: Aug 14 21:33:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:33:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:33:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:33:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:33:03 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 21:33:03 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 21:33:03 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:33:03 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:33:03 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:33:03 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:33:03 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:33:03 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:33:03 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:33:03 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:33:03 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:33:03 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:33:03 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:33:03 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:33:03 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:33:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:33:03 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:33:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:33:03 oak-gw06 kernel: Mem-Info: Aug 14 21:33:03 oak-gw06 kernel: active_anon:24816 inactive_anon:51096 isolated_anon:0#012 active_file:239263 inactive_file:1928383 isolated_file:0#012 unevictable:0 dirty:2943 writeback:2447 unstable:0#012 slab_reclaimable:35468 slab_unreclaimable:853392#012 mapped:10068 shmem:45078 pagetables:1691 bounce:0#012 free:837523 free_pcp:1577 free_cma:0 Aug 14 21:33:03 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:33:03 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:33:03 oak-gw06 kernel: Node 0 DMA32 free:739360kB min:69724kB low:87152kB high:104584kB active_anon:11296kB inactive_anon:35588kB active_file:186028kB inactive_file:1420748kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3292kB writeback:292kB mapped:4632kB shmem:31268kB slab_reclaimable:20684kB slab_unreclaimable:420468kB kernel_stack:944kB pagetables:1108kB unstable:0kB bounce:0kB free_pcp:3336kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:33:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:33:03 oak-gw06 kernel: Node 0 Normal free:2610140kB min:323104kB low:403880kB high:484656kB active_anon:87968kB inactive_anon:168796kB active_file:771024kB inactive_file:6276924kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8092kB writeback:7944kB mapped:35640kB shmem:149044kB slab_reclaimable:121188kB slab_unreclaimable:2993084kB kernel_stack:4768kB pagetables:5656kB unstable:0kB bounce:0kB free_pcp:3504kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:33:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:33:03 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:33:03 oak-gw06 kernel: Node 0 DMA32: 10958*4kB (UEM) 20154*8kB (UEM) 10023*16kB (UEM) 6444*32kB (UEM) 2251*64kB (UM) 145*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 734264kB Aug 14 21:33:03 oak-gw06 kernel: Node 0 Normal: 53799*4kB (UEM) 101648*8kB (UEM) 43132*16kB (UEM) 23225*32kB (UEM) 2348*64kB (UEM) 42*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2617340kB Aug 14 21:33:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:33:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:33:03 oak-gw06 kernel: 2089087 total pagecache pages Aug 14 21:33:03 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:33:03 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:33:03 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:33:03 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:33:03 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:33:03 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:33:03 oak-gw06 kernel: 127313 pages reserved Aug 14 21:33:03 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 21:33:03 oak-gw06 kernel: CPU: 6 PID: 3995 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:33:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:33:03 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:33:03 oak-gw06 kernel: 00000000000080d0 00000000b3e284da ffff8801df52b808 ffffffff8168662f Aug 14 21:33:03 oak-gw06 kernel: ffff8801df52b898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 21:33:03 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801df52b868 00000000b3e284da Aug 14 21:33:03 oak-gw06 kernel: Call Trace: Aug 14 21:33:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:33:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:33:03 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 21:33:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:33:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:33:03 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 21:33:03 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 21:33:03 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 21:33:03 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 21:33:03 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:33:03 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:33:03 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:33:03 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:33:03 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:33:03 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:33:03 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:33:03 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:33:03 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:33:03 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:33:03 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:33:03 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:33:03 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:33:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:33:03 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:33:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:33:03 oak-gw06 kernel: Mem-Info: Aug 14 21:33:03 oak-gw06 kernel: active_anon:24816 inactive_anon:51096 isolated_anon:0#012 active_file:239263 inactive_file:1930285 isolated_file:0#012 unevictable:0 dirty:3228 writeback:2065 unstable:0#012 slab_reclaimable:35468 slab_unreclaimable:853392#012 mapped:10068 shmem:45078 pagetables:1691 bounce:0#012 free:835320 free_pcp:1589 free_cma:0 Aug 14 21:33:03 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:33:03 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:33:03 oak-gw06 kernel: Node 0 DMA32 free:738300kB min:69724kB low:87152kB high:104584kB active_anon:11296kB inactive_anon:35588kB active_file:186028kB inactive_file:1426796kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4044kB writeback:0kB mapped:4632kB shmem:31268kB slab_reclaimable:20684kB slab_unreclaimable:420468kB kernel_stack:944kB pagetables:1108kB unstable:0kB bounce:0kB free_pcp:2752kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:33:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:33:03 oak-gw06 kernel: Node 0 Normal free:2580052kB min:323104kB low:403880kB high:484656kB active_anon:88488kB inactive_anon:168796kB active_file:771024kB inactive_file:6307864kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9256kB writeback:7556kB mapped:35640kB shmem:149044kB slab_reclaimable:121188kB slab_unreclaimable:2993084kB kernel_stack:4768kB pagetables:5656kB unstable:0kB bounce:0kB free_pcp:3372kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:33:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:33:03 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:33:03 oak-gw06 kernel: Node 0 DMA32: 10995*4kB (UEM) 19914*8kB (UEM) 10349*16kB (UEM) 6446*32kB (UEM) 2251*64kB (UM) 145*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 737772kB Aug 14 21:33:03 oak-gw06 kernel: Node 0 Normal: 48534*4kB (UEM) 99401*8kB (UEM) 42826*16kB (UEM) 23229*32kB (UEM) 2349*64kB (UEM) 42*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2573600kB Aug 14 21:33:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:33:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:33:03 oak-gw06 kernel: 2089820 total pagecache pages Aug 14 21:33:03 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:33:03 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:33:03 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:33:03 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:33:03 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:33:03 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:33:03 oak-gw06 kernel: 127313 pages reserved Aug 14 21:38:03 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 21:38:03 oak-gw06 kernel: CPU: 5 PID: 3981 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:38:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:38:03 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:38:03 oak-gw06 kernel: 00000000000080d0 00000000fc990a47 ffff8801969af858 ffffffff8168662f Aug 14 21:38:03 oak-gw06 kernel: ffff8801969af8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 21:38:03 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801969af8b8 00000000fc990a47 Aug 14 21:38:03 oak-gw06 kernel: Call Trace: Aug 14 21:38:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:38:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:38:03 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:38:03 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:38:03 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 21:38:03 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 21:38:03 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:38:03 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:38:03 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:38:03 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:38:03 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:38:03 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:38:03 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:38:03 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:38:03 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:38:03 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:38:03 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:38:03 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:38:03 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:38:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:38:03 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:38:03 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:38:03 oak-gw06 kernel: Mem-Info: Aug 14 21:38:03 oak-gw06 kernel: active_anon:24813 inactive_anon:51096 isolated_anon:0#012 active_file:115207 inactive_file:2028559 isolated_file:0#012 unevictable:0 dirty:2792 writeback:977 unstable:0#012 slab_reclaimable:35467 slab_unreclaimable:851147#012 mapped:10079 shmem:45078 pagetables:1688 bounce:0#012 free:866726 free_pcp:353 free_cma:0 Aug 14 21:38:03 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:38:03 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:38:03 oak-gw06 kernel: Node 0 DMA32 free:877036kB min:69724kB low:87152kB high:104584kB active_anon:10844kB inactive_anon:35588kB active_file:120652kB inactive_file:1351412kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3040kB writeback:132kB mapped:4632kB shmem:31268kB slab_reclaimable:20684kB slab_unreclaimable:417332kB kernel_stack:928kB pagetables:1084kB unstable:0kB bounce:0kB free_pcp:48kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:38:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:38:03 oak-gw06 kernel: Node 0 Normal free:2572196kB min:323104kB low:403880kB high:484656kB active_anon:88408kB inactive_anon:168796kB active_file:340176kB inactive_file:6763864kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8516kB writeback:2224kB mapped:35684kB shmem:149044kB slab_reclaimable:121184kB slab_unreclaimable:2987240kB kernel_stack:4752kB pagetables:5668kB unstable:0kB bounce:0kB free_pcp:1592kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:38:03 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:38:03 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:38:03 oak-gw06 kernel: Node 0 DMA32: 9675*4kB (UEM) 12749*8kB (UEM) 8116*16kB (UEM) 11195*32kB (UEM) 3388*64kB (UM) 256*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 878644kB Aug 14 21:38:03 oak-gw06 kernel: Node 0 Normal: 56929*4kB (UE) 96176*8kB (UEM) 31241*16kB (UEM) 25150*32kB (UEM) 3947*64kB (UEM) 123*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2570132kB Aug 14 21:38:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:38:03 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:38:03 oak-gw06 kernel: 2088347 total pagecache pages Aug 14 21:38:03 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:38:03 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:38:03 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:38:03 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:38:03 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:38:03 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:38:03 oak-gw06 kernel: 127313 pages reserved Aug 14 21:38:03 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 21:38:03 oak-gw06 kernel: CPU: 5 PID: 3981 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:38:03 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:38:03 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:38:03 oak-gw06 kernel: 00000000000080d0 00000000fc990a47 ffff8801969af808 ffffffff8168662f Aug 14 21:38:03 oak-gw06 kernel: ffff8801969af898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 21:38:03 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801969af868 00000000fc990a47 Aug 14 21:38:03 oak-gw06 kernel: Call Trace: Aug 14 21:38:03 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:38:03 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:38:04 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 21:38:04 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:38:04 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:38:04 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 21:38:04 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 21:38:04 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 21:38:04 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 21:38:04 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:38:04 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:38:04 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:38:04 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:38:04 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:38:04 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:38:04 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:38:04 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:38:04 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:38:04 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:38:04 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:38:04 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:38:04 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:38:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:38:04 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:38:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:38:04 oak-gw06 kernel: Mem-Info: Aug 14 21:38:04 oak-gw06 kernel: active_anon:24813 inactive_anon:51096 isolated_anon:0#012 active_file:115272 inactive_file:2030509 isolated_file:0#012 unevictable:0 dirty:3277 writeback:589 unstable:0#012 slab_reclaimable:35467 slab_unreclaimable:851147#012 mapped:10079 shmem:45078 pagetables:1688 bounce:0#012 free:864690 free_pcp:404 free_cma:0 Aug 14 21:38:04 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:38:04 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:38:04 oak-gw06 kernel: Node 0 DMA32 free:878048kB min:69724kB low:87152kB high:104584kB active_anon:10844kB inactive_anon:35588kB active_file:120652kB inactive_file:1351412kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3040kB writeback:132kB mapped:4632kB shmem:31268kB slab_reclaimable:20684kB slab_unreclaimable:417332kB kernel_stack:928kB pagetables:1084kB unstable:0kB bounce:0kB free_pcp:32kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:38:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:38:04 oak-gw06 kernel: Node 0 Normal free:2562948kB min:323104kB low:403880kB high:484656kB active_anon:88408kB inactive_anon:168796kB active_file:340436kB inactive_file:6772184kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9680kB writeback:1448kB mapped:35684kB shmem:149044kB slab_reclaimable:121184kB slab_unreclaimable:2987240kB kernel_stack:4752kB pagetables:5668kB unstable:0kB bounce:0kB free_pcp:2032kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:38:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:38:04 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:38:04 oak-gw06 kernel: Node 0 DMA32: 9833*4kB (UEM) 12749*8kB (UEM) 8166*16kB (UEM) 11195*32kB (UEM) 3388*64kB (UM) 256*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 880076kB Aug 14 21:38:04 oak-gw06 kernel: Node 0 Normal: 56949*4kB (UEM) 95085*8kB (UEM) 31173*16kB (UEM) 25150*32kB (UEM) 3947*64kB (UEM) 123*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2560396kB Aug 14 21:38:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:38:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:38:04 oak-gw06 kernel: 2088365 total pagecache pages Aug 14 21:38:04 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:38:04 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:38:04 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:38:04 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:38:04 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:38:04 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:38:04 oak-gw06 kernel: 127313 pages reserved Aug 14 21:43:04 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 21:43:04 oak-gw06 kernel: CPU: 6 PID: 4010 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:43:04 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:43:04 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:43:04 oak-gw06 kernel: 00000000000080d0 00000000b8ee7963 ffff880361ab7858 ffffffff8168662f Aug 14 21:43:04 oak-gw06 kernel: ffff880361ab78e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 21:43:04 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880361ab78b8 00000000b8ee7963 Aug 14 21:43:04 oak-gw06 kernel: Call Trace: Aug 14 21:43:04 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:43:04 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:43:04 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:43:04 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:43:04 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 21:43:04 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 21:43:04 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:43:04 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:43:04 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:43:04 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:43:04 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:43:04 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:43:04 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:43:04 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:43:04 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:43:04 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:43:04 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:43:04 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:43:04 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:43:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:43:04 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:43:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:43:04 oak-gw06 kernel: Mem-Info: Aug 14 21:43:04 oak-gw06 kernel: active_anon:24820 inactive_anon:51096 isolated_anon:0#012 active_file:199739 inactive_file:2217675 isolated_file:0#012 unevictable:0 dirty:3251 writeback:639 unstable:0#012 slab_reclaimable:35467 slab_unreclaimable:852042#012 mapped:10090 shmem:45078 pagetables:1688 bounce:0#012 free:576288 free_pcp:1102 free_cma:0 Aug 14 21:43:04 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:43:04 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:43:04 oak-gw06 kernel: Node 0 DMA32 free:669188kB min:69724kB low:87152kB high:104584kB active_anon:10844kB inactive_anon:35588kB active_file:166352kB inactive_file:1492328kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2200kB writeback:292kB mapped:4632kB shmem:31268kB slab_reclaimable:20684kB slab_unreclaimable:424896kB kernel_stack:912kB pagetables:1080kB unstable:0kB bounce:0kB free_pcp:1576kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:43:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:43:04 oak-gw06 kernel: Node 0 Normal free:1612892kB min:323104kB low:403880kB high:484656kB active_anon:88436kB inactive_anon:168796kB active_file:632604kB inactive_file:7384352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9252kB writeback:6920kB mapped:35728kB shmem:149044kB slab_reclaimable:121184kB slab_unreclaimable:2983256kB kernel_stack:4784kB pagetables:5672kB unstable:0kB bounce:0kB free_pcp:3936kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:43:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:43:04 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:43:04 oak-gw06 kernel: Node 0 DMA32: 9595*4kB (UEM) 8245*8kB (UEM) 730*16kB (UEM) 8283*32kB (UEM) 3798*64kB (UM) 373*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 672404kB Aug 14 21:43:04 oak-gw06 kernel: Node 0 Normal: 45983*4kB (UE) 40960*8kB (UE) 5891*16kB (UE) 24765*32kB (UEM) 3210*64kB (UEM) 63*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1611852kB Aug 14 21:43:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:43:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:43:04 oak-gw06 kernel: 2088919 total pagecache pages Aug 14 21:43:04 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:43:04 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:43:04 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:43:04 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:43:04 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:43:04 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:43:04 oak-gw06 kernel: 127313 pages reserved Aug 14 21:43:04 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 21:43:04 oak-gw06 kernel: CPU: 6 PID: 4010 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:43:04 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:43:04 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:43:04 oak-gw06 kernel: 00000000000080d0 00000000b8ee7963 ffff880361ab7808 ffffffff8168662f Aug 14 21:43:04 oak-gw06 kernel: ffff880361ab7898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 21:43:04 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880361ab7868 00000000b8ee7963 Aug 14 21:43:04 oak-gw06 kernel: Call Trace: Aug 14 21:43:04 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:43:04 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:43:04 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 21:43:04 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:43:04 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:43:04 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 21:43:04 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 21:43:04 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 21:43:04 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 21:43:04 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:43:04 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:43:04 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:43:04 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:43:04 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:43:04 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:43:04 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:43:04 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:43:04 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:43:04 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:43:04 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:43:04 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:43:04 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:43:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:43:04 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:43:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:43:04 oak-gw06 kernel: Mem-Info: Aug 14 21:43:04 oak-gw06 kernel: active_anon:24820 inactive_anon:51096 isolated_anon:0#012 active_file:199674 inactive_file:2222955 isolated_file:0#012 unevictable:0 dirty:2863 writeback:1706 unstable:0#012 slab_reclaimable:35467 slab_unreclaimable:852042#012 mapped:10090 shmem:45078 pagetables:1688 bounce:0#012 free:571774 free_pcp:1129 free_cma:0 Aug 14 21:43:04 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:43:04 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:43:04 oak-gw06 kernel: Node 0 DMA32 free:678960kB min:69724kB low:87152kB high:104584kB active_anon:10844kB inactive_anon:35588kB active_file:166352kB inactive_file:1491348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2200kB writeback:292kB mapped:4632kB shmem:31268kB slab_reclaimable:20684kB slab_unreclaimable:424896kB kernel_stack:912kB pagetables:1080kB unstable:0kB bounce:0kB free_pcp:1308kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:43:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:43:04 oak-gw06 kernel: Node 0 Normal free:1586504kB min:323104kB low:403880kB high:484656kB active_anon:88436kB inactive_anon:168796kB active_file:632344kB inactive_file:7412980kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:7856kB writeback:11964kB mapped:35728kB shmem:149044kB slab_reclaimable:121184kB slab_unreclaimable:2983432kB kernel_stack:4784kB pagetables:5672kB unstable:0kB bounce:0kB free_pcp:4364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:43:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:43:04 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:43:04 oak-gw06 kernel: Node 0 DMA32: 8071*4kB (UEM) 8165*8kB (UEM) 653*16kB (UE) 8211*32kB (UEM) 3798*64kB (UM) 373*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 662132kB Aug 14 21:43:04 oak-gw06 kernel: Node 0 Normal: 47598*4kB (UEM) 41083*8kB (UE) 6249*16kB (UE) 23681*32kB (UEM) 3225*64kB (UEM) 63*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1591296kB Aug 14 21:43:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:43:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:43:04 oak-gw06 kernel: 2090186 total pagecache pages Aug 14 21:43:04 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:43:04 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:43:04 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:43:04 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:43:04 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:43:04 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:43:04 oak-gw06 kernel: 127313 pages reserved Aug 14 21:48:04 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 21:48:04 oak-gw06 kernel: CPU: 7 PID: 4019 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:48:04 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:48:04 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:48:04 oak-gw06 kernel: 00000000000080d0 00000000cfca4a15 ffff88008be57858 ffffffff8168662f Aug 14 21:48:04 oak-gw06 kernel: ffff88008be578e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 21:48:04 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88008be578b8 00000000cfca4a15 Aug 14 21:48:04 oak-gw06 kernel: Call Trace: Aug 14 21:48:04 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:48:04 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:48:04 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:48:04 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:48:04 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 21:48:04 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 21:48:04 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:48:04 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:48:04 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:48:04 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:48:04 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:48:04 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:48:04 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:48:04 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:48:04 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:48:04 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:48:04 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:48:04 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:48:04 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:48:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:48:04 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:48:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:48:04 oak-gw06 kernel: Mem-Info: Aug 14 21:48:04 oak-gw06 kernel: active_anon:20437 inactive_anon:51096 isolated_anon:0#012 active_file:328988 inactive_file:1862443 isolated_file:0#012 unevictable:0 dirty:2300 writeback:4876 unstable:0#012 slab_reclaimable:35390 slab_unreclaimable:839735#012 mapped:10091 shmem:45078 pagetables:1678 bounce:0#012 free:833777 free_pcp:983 free_cma:0 Aug 14 21:48:04 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:48:04 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:48:04 oak-gw06 kernel: Node 0 DMA32 free:899668kB min:69724kB low:87152kB high:104584kB active_anon:10128kB inactive_anon:35588kB active_file:242468kB inactive_file:1207732kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1632kB writeback:1708kB mapped:4644kB shmem:31268kB slab_reclaimable:20644kB slab_unreclaimable:420108kB kernel_stack:928kB pagetables:1096kB unstable:0kB bounce:0kB free_pcp:628kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:48:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:48:04 oak-gw06 kernel: Node 0 Normal free:2478492kB min:323104kB low:403880kB high:484656kB active_anon:71620kB inactive_anon:168796kB active_file:1073484kB inactive_file:6182592kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9508kB writeback:20512kB mapped:35720kB shmem:149044kB slab_reclaimable:120916kB slab_unreclaimable:2938816kB kernel_stack:4768kB pagetables:5616kB unstable:0kB bounce:0kB free_pcp:3852kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:48:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:48:04 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:48:04 oak-gw06 kernel: Node 0 DMA32: 14291*4kB (UEM) 24225*8kB (UEM) 15940*16kB (UEM) 5912*32kB (UEM) 2710*64kB (UM) 275*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 904340kB Aug 14 21:48:04 oak-gw06 kernel: Node 0 Normal: 53764*4kB (UEM) 91952*8kB (UEM) 62002*16kB (UEM) 14824*32kB (UEM) 949*64kB (UM) 26*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2481136kB Aug 14 21:48:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:48:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:48:04 oak-gw06 kernel: 2086018 total pagecache pages Aug 14 21:48:04 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:48:04 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:48:04 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:48:04 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:48:04 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:48:04 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:48:04 oak-gw06 kernel: 127313 pages reserved Aug 14 21:48:04 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 21:48:04 oak-gw06 kernel: CPU: 7 PID: 4019 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:48:04 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:48:04 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:48:04 oak-gw06 kernel: 00000000000080d0 00000000cfca4a15 ffff88008be57808 ffffffff8168662f Aug 14 21:48:04 oak-gw06 kernel: ffff88008be57898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 21:48:04 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88008be57868 00000000cfca4a15 Aug 14 21:48:04 oak-gw06 kernel: Call Trace: Aug 14 21:48:04 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:48:04 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:48:04 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 21:48:04 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:48:04 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:48:04 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 21:48:04 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 21:48:04 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 21:48:04 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 21:48:04 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:48:04 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:48:04 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:48:04 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:48:04 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:48:04 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:48:04 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:48:04 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:48:04 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:48:04 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:48:04 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:48:04 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:48:04 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:48:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:48:04 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:48:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:48:04 oak-gw06 kernel: Mem-Info: Aug 14 21:48:04 oak-gw06 kernel: active_anon:20437 inactive_anon:51096 isolated_anon:0#012 active_file:328988 inactive_file:1849937 isolated_file:0#012 unevictable:0 dirty:2494 writeback:2748 unstable:0#012 slab_reclaimable:35390 slab_unreclaimable:839735#012 mapped:10091 shmem:45078 pagetables:1678 bounce:0#012 free:847381 free_pcp:424 free_cma:0 Aug 14 21:48:04 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:48:04 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:48:04 oak-gw06 kernel: Node 0 DMA32 free:906264kB min:69724kB low:87152kB high:104584kB active_anon:10128kB inactive_anon:35588kB active_file:242468kB inactive_file:1204708kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1632kB writeback:956kB mapped:4644kB shmem:31268kB slab_reclaimable:20644kB slab_unreclaimable:420108kB kernel_stack:928kB pagetables:1096kB unstable:0kB bounce:0kB free_pcp:668kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:48:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:48:04 oak-gw06 kernel: Node 0 Normal free:2463952kB min:323104kB low:403880kB high:484656kB active_anon:72400kB inactive_anon:168796kB active_file:1073484kB inactive_file:6197412kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9120kB writeback:10812kB mapped:35720kB shmem:149044kB slab_reclaimable:120916kB slab_unreclaimable:2938816kB kernel_stack:4768kB pagetables:5616kB unstable:0kB bounce:0kB free_pcp:1604kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:48:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:48:04 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:48:04 oak-gw06 kernel: Node 0 DMA32: 14982*4kB (UEM) 24241*8kB (UEM) 16125*16kB (UEM) 5915*32kB (UEM) 2710*64kB (UM) 275*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 910288kB Aug 14 21:48:04 oak-gw06 kernel: Node 0 Normal: 48435*4kB (UEM) 91969*8kB (UEM) 61771*16kB (UEM) 14824*32kB (UEM) 949*64kB (UM) 26*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2456260kB Aug 14 21:48:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:48:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:48:04 oak-gw06 kernel: 2087879 total pagecache pages Aug 14 21:48:04 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:48:04 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:48:04 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:48:04 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:48:04 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:48:04 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:48:04 oak-gw06 kernel: 127313 pages reserved Aug 14 21:53:04 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 21:53:04 oak-gw06 kernel: CPU: 6 PID: 4019 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:53:04 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:53:04 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:53:04 oak-gw06 kernel: 00000000000080d0 00000000cfca4a15 ffff88008be57858 ffffffff8168662f Aug 14 21:53:04 oak-gw06 kernel: ffff88008be578e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 21:53:04 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88008be578b8 00000000cfca4a15 Aug 14 21:53:04 oak-gw06 kernel: Call Trace: Aug 14 21:53:04 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:53:04 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:53:04 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:53:04 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:53:04 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 21:53:04 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 21:53:04 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:53:04 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:53:04 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:53:04 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:53:04 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:53:04 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:53:04 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:53:04 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:53:04 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:53:04 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:53:04 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:53:04 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:53:04 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:53:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:53:04 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:53:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:53:04 oak-gw06 kernel: Mem-Info: Aug 14 21:53:04 oak-gw06 kernel: active_anon:25785 inactive_anon:51096 isolated_anon:0#012 active_file:190755 inactive_file:2366569 isolated_file:0#012 unevictable:0 dirty:8950 writeback:2424 unstable:0#012 slab_reclaimable:35126 slab_unreclaimable:826944#012 mapped:10114 shmem:45078 pagetables:1690 bounce:0#012 free:444707 free_pcp:253 free_cma:0 Aug 14 21:53:04 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:53:04 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:53:04 oak-gw06 kernel: Node 0 DMA32 free:456752kB min:69724kB low:87152kB high:104584kB active_anon:10008kB inactive_anon:35588kB active_file:168688kB inactive_file:1699692kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:6360kB writeback:120kB mapped:4636kB shmem:31268kB slab_reclaimable:20532kB slab_unreclaimable:418928kB kernel_stack:928kB pagetables:1100kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:53:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:53:04 oak-gw06 kernel: Node 0 Normal free:1310068kB min:323104kB low:403880kB high:484656kB active_anon:93132kB inactive_anon:168796kB active_file:594332kB inactive_file:7760344kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:29440kB writeback:9576kB mapped:35820kB shmem:149044kB slab_reclaimable:119972kB slab_unreclaimable:2888832kB kernel_stack:4768kB pagetables:5660kB unstable:0kB bounce:0kB free_pcp:3524kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:53:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:53:04 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:53:04 oak-gw06 kernel: Node 0 DMA32: 7092*4kB (UEM) 8119*8kB (UEM) 890*16kB (UEM) 4221*32kB (UEM) 2892*64kB (UM) 264*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 462024kB Aug 14 21:53:04 oak-gw06 kernel: Node 0 Normal: 34333*4kB (UEM) 36130*8kB (UE) 6201*16kB (UEM) 20938*32kB (UEM) 1904*64kB (UM) 13*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1319124kB Aug 14 21:53:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:53:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:53:04 oak-gw06 kernel: 2080838 total pagecache pages Aug 14 21:53:04 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:53:04 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:53:04 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:53:04 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:53:04 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:53:04 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:53:04 oak-gw06 kernel: 127313 pages reserved Aug 14 21:53:04 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 21:53:04 oak-gw06 kernel: CPU: 6 PID: 4019 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:53:04 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:53:04 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:53:04 oak-gw06 kernel: 00000000000080d0 00000000cfca4a15 ffff88008be57808 ffffffff8168662f Aug 14 21:53:04 oak-gw06 kernel: ffff88008be57898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 21:53:04 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88008be57868 00000000cfca4a15 Aug 14 21:53:04 oak-gw06 kernel: Call Trace: Aug 14 21:53:04 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:53:04 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:53:04 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:53:04 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:53:04 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 21:53:04 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 21:53:04 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 21:53:04 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 21:53:04 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:53:04 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:53:04 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:53:04 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:53:04 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:53:04 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:53:04 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:53:04 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:53:04 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:53:04 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:53:04 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:53:04 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:53:04 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:53:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:53:04 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:53:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:53:04 oak-gw06 kernel: Mem-Info: Aug 14 21:53:04 oak-gw06 kernel: active_anon:25785 inactive_anon:51096 isolated_anon:0#012 active_file:190629 inactive_file:2373235 isolated_file:0#012 unevictable:0 dirty:8950 writeback:2424 unstable:0#012 slab_reclaimable:35126 slab_unreclaimable:826944#012 mapped:10114 shmem:45078 pagetables:1690 bounce:0#012 free:438705 free_pcp:69 free_cma:0 Aug 14 21:53:04 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:53:04 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:53:04 oak-gw06 kernel: Node 0 DMA32 free:462808kB min:69724kB low:87152kB high:104584kB active_anon:10008kB inactive_anon:35588kB active_file:168184kB inactive_file:1695156kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:6360kB writeback:120kB mapped:4636kB shmem:31268kB slab_reclaimable:20532kB slab_unreclaimable:418928kB kernel_stack:928kB pagetables:1100kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:53:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:53:04 oak-gw06 kernel: Node 0 Normal free:1260088kB min:323104kB low:403880kB high:484656kB active_anon:93132kB inactive_anon:168796kB active_file:594332kB inactive_file:7812084kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:29440kB writeback:9576kB mapped:35820kB shmem:149044kB slab_reclaimable:119972kB slab_unreclaimable:2889376kB kernel_stack:4768kB pagetables:5660kB unstable:0kB bounce:0kB free_pcp:736kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:53:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:53:04 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:53:04 oak-gw06 kernel: Node 0 DMA32: 7217*4kB (UEM) 8118*8kB (UEM) 979*16kB (UEM) 4221*32kB (UEM) 2892*64kB (UM) 264*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 463940kB Aug 14 21:53:04 oak-gw06 kernel: Node 0 Normal: 34331*4kB (UEM) 36131*8kB (UEM) 3051*16kB (UEM) 20569*32kB (UEM) 1904*64kB (UM) 13*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1256916kB Aug 14 21:53:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:53:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:53:04 oak-gw06 kernel: 2090544 total pagecache pages Aug 14 21:53:04 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:53:04 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:53:04 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:53:04 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:53:04 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:53:04 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:53:04 oak-gw06 kernel: 127313 pages reserved Aug 14 21:58:04 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 21:58:04 oak-gw06 kernel: CPU: 6 PID: 4034 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:58:04 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:58:04 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:58:04 oak-gw06 kernel: 00000000000080d0 00000000e2883427 ffff880205bff858 ffffffff8168662f Aug 14 21:58:04 oak-gw06 kernel: ffff880205bff8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 21:58:04 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880205bff8b8 00000000e2883427 Aug 14 21:58:04 oak-gw06 kernel: Call Trace: Aug 14 21:58:04 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:58:04 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:58:04 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:58:04 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:58:04 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 21:58:04 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 21:58:04 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:58:04 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:58:04 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:58:04 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:58:04 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:58:04 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:58:04 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:58:04 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:58:04 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:58:04 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:58:04 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:58:04 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:58:04 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:58:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:58:04 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:58:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:58:04 oak-gw06 kernel: Mem-Info: Aug 14 21:58:04 oak-gw06 kernel: active_anon:16323 inactive_anon:51096 isolated_anon:0#012 active_file:371944 inactive_file:1801784 isolated_file:0#012 unevictable:0 dirty:2474 writeback:440 unstable:0#012 slab_reclaimable:34922 slab_unreclaimable:799969#012 mapped:10125 shmem:45078 pagetables:1665 bounce:0#012 free:896328 free_pcp:290 free_cma:0 Aug 14 21:58:04 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:58:04 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:58:04 oak-gw06 kernel: Node 0 DMA32 free:762548kB min:69724kB low:87152kB high:104584kB active_anon:14872kB inactive_anon:35588kB active_file:300632kB inactive_file:1272928kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1900kB writeback:292kB mapped:4660kB shmem:31268kB slab_reclaimable:20424kB slab_unreclaimable:420720kB kernel_stack:976kB pagetables:1968kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:58:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:58:04 oak-gw06 kernel: Node 0 Normal free:2805568kB min:323104kB low:403880kB high:484656kB active_anon:50420kB inactive_anon:168796kB active_file:1187144kB inactive_file:5935248kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8384kB writeback:1468kB mapped:35840kB shmem:149044kB slab_reclaimable:119264kB slab_unreclaimable:2779140kB kernel_stack:4784kB pagetables:4692kB unstable:0kB bounce:0kB free_pcp:1252kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:58:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:58:04 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:58:04 oak-gw06 kernel: Node 0 DMA32: 10747*4kB (UEM) 17654*8kB (UEM) 8030*16kB (UEM) 8537*32kB (UEM) 2566*64kB (UM) 105*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 764060kB Aug 14 21:58:04 oak-gw06 kernel: Node 0 Normal: 55622*4kB (UEM) 92325*8kB (UEM) 43604*16kB (UEM) 29374*32kB (UEM) 3071*64kB (UEM) 55*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2802304kB Aug 14 21:58:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:58:04 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:58:04 oak-gw06 kernel: 2079133 total pagecache pages Aug 14 21:58:04 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:58:04 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:58:04 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:58:04 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:58:04 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:58:04 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:58:04 oak-gw06 kernel: 127313 pages reserved Aug 14 21:58:04 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 21:58:04 oak-gw06 kernel: CPU: 0 PID: 4034 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 21:58:04 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 21:58:04 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 21:58:04 oak-gw06 kernel: 00000000000080d0 00000000e2883427 ffff880205bff808 ffffffff8168662f Aug 14 21:58:04 oak-gw06 kernel: ffff880205bff898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 21:58:04 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880205bff868 00000000e2883427 Aug 14 21:58:04 oak-gw06 kernel: Call Trace: Aug 14 21:58:04 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 21:58:04 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 21:58:04 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 21:58:04 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 21:58:04 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 21:58:04 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 21:58:04 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 21:58:04 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 21:58:04 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 21:58:04 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 21:58:04 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 21:58:04 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 21:58:04 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 21:58:04 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 21:58:04 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 21:58:04 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 21:58:04 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 21:58:04 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 21:58:04 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 21:58:04 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 21:58:04 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 21:58:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:58:04 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 21:58:04 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 21:58:04 oak-gw06 kernel: Mem-Info: Aug 14 21:58:04 oak-gw06 kernel: active_anon:16388 inactive_anon:51096 isolated_anon:0#012 active_file:371944 inactive_file:1805294 isolated_file:0#012 unevictable:0 dirty:2377 writeback:1313 unstable:0#012 slab_reclaimable:34922 slab_unreclaimable:799969#012 mapped:10125 shmem:45078 pagetables:1665 bounce:0#012 free:892530 free_pcp:552 free_cma:0 Aug 14 21:58:04 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 21:58:04 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 21:58:04 oak-gw06 kernel: Node 0 DMA32 free:768172kB min:69724kB low:87152kB high:104584kB active_anon:14872kB inactive_anon:35588kB active_file:300632kB inactive_file:1272928kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1900kB writeback:292kB mapped:4660kB shmem:31268kB slab_reclaimable:20424kB slab_unreclaimable:420720kB kernel_stack:976kB pagetables:1968kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:58:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 21:58:04 oak-gw06 kernel: Node 0 Normal free:2776772kB min:323104kB low:403880kB high:484656kB active_anon:50940kB inactive_anon:168796kB active_file:1187144kB inactive_file:5954488kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:7608kB writeback:3796kB mapped:35840kB shmem:149044kB slab_reclaimable:119264kB slab_unreclaimable:2779140kB kernel_stack:4784kB pagetables:4692kB unstable:0kB bounce:0kB free_pcp:2500kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 21:58:04 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 21:58:04 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 21:58:04 oak-gw06 kernel: Node 0 DMA32: 10747*4kB (UEM) 17654*8kB (UEM) 8518*16kB (UEM) 8537*32kB (UEM) 2566*64kB (UM) 105*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 771868kB Aug 14 21:58:05 oak-gw06 kernel: Node 0 Normal: 55546*4kB (UEM) 89355*8kB (UEM) 43174*16kB (UEM) 29374*32kB (UEM) 3071*64kB (UEM) 55*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2771360kB Aug 14 21:58:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 21:58:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 21:58:05 oak-gw06 kernel: 2085341 total pagecache pages Aug 14 21:58:05 oak-gw06 kernel: 16 pages in swap cache Aug 14 21:58:05 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 21:58:05 oak-gw06 kernel: Free swap = 4194036kB Aug 14 21:58:05 oak-gw06 kernel: Total swap = 4194300kB Aug 14 21:58:05 oak-gw06 kernel: 4194203 pages RAM Aug 14 21:58:05 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 21:58:05 oak-gw06 kernel: 127313 pages reserved Aug 14 22:03:04 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 22:03:04 oak-gw06 kernel: CPU: 4 PID: 4019 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:03:04 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:03:04 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:03:04 oak-gw06 kernel: 00000000000080d0 00000000cfca4a15 ffff88008be57858 ffffffff8168662f Aug 14 22:03:04 oak-gw06 kernel: ffff88008be578e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:03:04 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88008be578b8 00000000cfca4a15 Aug 14 22:03:04 oak-gw06 kernel: Call Trace: Aug 14 22:03:04 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:03:04 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:03:04 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:03:04 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:03:04 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 22:03:04 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 22:03:04 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:03:04 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:03:04 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:03:04 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:03:04 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:03:04 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:03:05 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:03:05 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:03:05 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:03:05 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:03:05 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:03:05 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:03:05 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:03:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:03:05 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:03:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:03:05 oak-gw06 kernel: Mem-Info: Aug 14 22:03:05 oak-gw06 kernel: active_anon:24807 inactive_anon:51096 isolated_anon:0#012 active_file:239524 inactive_file:1865644 isolated_file:0#012 unevictable:0 dirty:2550 writeback:4281 unstable:0#012 slab_reclaimable:34914 slab_unreclaimable:799058#012 mapped:10153 shmem:45078 pagetables:1694 bounce:0#012 free:934815 free_pcp:952 free_cma:0 Aug 14 22:03:05 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:03:05 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:03:05 oak-gw06 kernel: Node 0 DMA32 free:773816kB min:69724kB low:87152kB high:104584kB active_anon:16008kB inactive_anon:35588kB active_file:201660kB inactive_file:1349652kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1704kB writeback:3140kB mapped:4648kB shmem:31268kB slab_reclaimable:20424kB slab_unreclaimable:418720kB kernel_stack:1008kB pagetables:1980kB unstable:0kB bounce:0kB free_pcp:712kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:03:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:03:05 oak-gw06 kernel: Node 0 Normal free:2946404kB min:323104kB low:403880kB high:484656kB active_anon:83480kB inactive_anon:168796kB active_file:756436kB inactive_file:6118124kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8496kB writeback:9812kB mapped:35964kB shmem:149044kB slab_reclaimable:119232kB slab_unreclaimable:2777496kB kernel_stack:4720kB pagetables:4796kB unstable:0kB bounce:0kB free_pcp:3408kB local_pcp:116kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:03:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:03:05 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:03:05 oak-gw06 kernel: Node 0 DMA32: 7848*4kB (UEM) 11943*8kB (UEM) 11397*16kB (UEM) 7995*32kB (UEM) 2857*64kB (UM) 214*128kB (UM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 776392kB Aug 14 22:03:05 oak-gw06 kernel: Node 0 Normal: 50476*4kB (UE) 61165*8kB (UEM) 61263*16kB (UEM) 31142*32kB (UEM) 3925*64kB (UEM) 114*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2933768kB Aug 14 22:03:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:03:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:03:05 oak-gw06 kernel: 2079242 total pagecache pages Aug 14 22:03:05 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:03:05 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:03:05 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:03:05 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:03:05 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:03:05 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:03:05 oak-gw06 kernel: 127313 pages reserved Aug 14 22:03:05 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 22:03:05 oak-gw06 kernel: CPU: 0 PID: 4019 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:03:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:03:05 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:03:05 oak-gw06 kernel: 00000000000080d0 00000000cfca4a15 ffff88008be57808 ffffffff8168662f Aug 14 22:03:05 oak-gw06 kernel: ffff88008be57898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:03:05 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88008be57868 00000000cfca4a15 Aug 14 22:03:05 oak-gw06 kernel: Call Trace: Aug 14 22:03:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:03:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:03:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:03:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:03:05 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 22:03:05 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 22:03:05 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 22:03:05 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 22:03:05 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:03:05 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:03:05 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:03:05 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:03:05 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:03:05 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:03:05 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:03:05 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:03:05 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:03:05 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:03:05 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:03:05 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:03:05 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:03:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:03:05 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:03:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:03:05 oak-gw06 kernel: Mem-Info: Aug 14 22:03:05 oak-gw06 kernel: active_anon:24807 inactive_anon:51096 isolated_anon:0#012 active_file:239524 inactive_file:1877180 isolated_file:0#012 unevictable:0 dirty:3010 writeback:2294 unstable:0#012 slab_reclaimable:34914 slab_unreclaimable:798744#012 mapped:10153 shmem:45078 pagetables:1694 bounce:0#012 free:929382 free_pcp:566 free_cma:0 Aug 14 22:03:05 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:03:05 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:03:05 oak-gw06 kernel: Node 0 DMA32 free:780532kB min:69724kB low:87152kB high:104584kB active_anon:16008kB inactive_anon:35588kB active_file:201660kB inactive_file:1349628kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1824kB writeback:112kB mapped:4648kB shmem:31268kB slab_reclaimable:20424kB slab_unreclaimable:418784kB kernel_stack:1008kB pagetables:1980kB unstable:0kB bounce:0kB free_pcp:628kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:03:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:03:05 oak-gw06 kernel: Node 0 Normal free:2914036kB min:323104kB low:403880kB high:484656kB active_anon:83480kB inactive_anon:168796kB active_file:756436kB inactive_file:6170272kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9828kB writeback:7124kB mapped:35964kB shmem:149044kB slab_reclaimable:119232kB slab_unreclaimable:2776176kB kernel_stack:4720kB pagetables:4796kB unstable:0kB bounce:0kB free_pcp:1464kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:03:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:03:05 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:03:05 oak-gw06 kernel: Node 0 DMA32: 8463*4kB (UEM) 11946*8kB (UEM) 11640*16kB (UEM) 7999*32kB (UEM) 2857*64kB (UM) 214*128kB (UM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 782892kB Aug 14 22:03:05 oak-gw06 kernel: Node 0 Normal: 53040*4kB (UEM) 55297*8kB (UEM) 61914*16kB (UEM) 31146*32kB (UEM) 3925*64kB (UEM) 114*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2907624kB Aug 14 22:03:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:03:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:03:05 oak-gw06 kernel: 2090861 total pagecache pages Aug 14 22:03:05 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:03:05 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:03:05 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:03:05 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:03:05 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:03:05 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:03:05 oak-gw06 kernel: 127313 pages reserved Aug 14 22:08:05 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 22:08:05 oak-gw06 kernel: CPU: 6 PID: 4047 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:08:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:08:05 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:08:05 oak-gw06 kernel: 00000000000080d0 0000000072942ab0 ffff880036c77858 ffffffff8168662f Aug 14 22:08:05 oak-gw06 kernel: ffff880036c778e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:08:05 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880036c778b8 0000000072942ab0 Aug 14 22:08:05 oak-gw06 kernel: Call Trace: Aug 14 22:08:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:08:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:08:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:08:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:08:05 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 22:08:05 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 22:08:05 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:08:05 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:08:05 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:08:05 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:08:05 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:08:05 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:08:05 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:08:05 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:08:05 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:08:05 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:08:05 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:08:05 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:08:05 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:08:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:08:05 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:08:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:08:05 oak-gw06 kernel: Mem-Info: Aug 14 22:08:05 oak-gw06 kernel: active_anon:24810 inactive_anon:51096 isolated_anon:0#012 active_file:389904 inactive_file:2009632 isolated_file:0#012 unevictable:0 dirty:2530 writeback:3466 unstable:0#012 slab_reclaimable:34736 slab_unreclaimable:787625#012 mapped:10282 shmem:45078 pagetables:1695 bounce:0#012 free:628107 free_pcp:851 free_cma:0 Aug 14 22:08:05 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:08:05 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:08:05 oak-gw06 kernel: Node 0 DMA32 free:549364kB min:69724kB low:87152kB high:104584kB active_anon:16876kB inactive_anon:35588kB active_file:361180kB inactive_file:1408816kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:108kB writeback:0kB mapped:4712kB shmem:31268kB slab_reclaimable:20360kB slab_unreclaimable:415888kB kernel_stack:1008kB pagetables:2104kB unstable:0kB bounce:0kB free_pcp:1428kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:08:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:08:05 oak-gw06 kernel: Node 0 Normal free:1949112kB min:323104kB low:403880kB high:484656kB active_anon:82364kB inactive_anon:168796kB active_file:1198436kB inactive_file:6641152kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9236kB writeback:15448kB mapped:36416kB shmem:149044kB slab_reclaimable:118584kB slab_unreclaimable:2734596kB kernel_stack:4672kB pagetables:4676kB unstable:0kB bounce:0kB free_pcp:3308kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:08:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:08:05 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:08:05 oak-gw06 kernel: Node 0 DMA32: 4611*4kB (UEM) 7545*8kB (UEM) 8504*16kB (UEM) 6575*32kB (UEM) 1677*64kB (UM) 29*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 536564kB Aug 14 22:08:05 oak-gw06 kernel: Node 0 Normal: 37993*4kB (UE) 57006*8kB (UEM) 31107*16kB (UEM) 20728*32kB (UEM) 2935*64kB (UEM) 60*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1964548kB Aug 14 22:08:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:08:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:08:05 oak-gw06 kernel: 2087493 total pagecache pages Aug 14 22:08:05 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:08:05 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:08:05 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:08:05 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:08:05 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:08:05 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:08:05 oak-gw06 kernel: 127313 pages reserved Aug 14 22:08:05 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 22:08:05 oak-gw06 kernel: CPU: 6 PID: 4047 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:08:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:08:05 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:08:05 oak-gw06 kernel: 00000000000080d0 0000000072942ab0 ffff880036c77808 ffffffff8168662f Aug 14 22:08:05 oak-gw06 kernel: ffff880036c77898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:08:05 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880036c77868 0000000072942ab0 Aug 14 22:08:05 oak-gw06 kernel: Call Trace: Aug 14 22:08:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:08:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:08:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:08:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:08:05 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 22:08:05 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 22:08:05 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 22:08:05 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 22:08:05 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:08:05 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:08:05 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:08:05 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:08:05 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:08:05 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:08:05 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:08:05 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:08:05 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:08:05 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:08:05 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:08:05 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:08:05 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:08:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:08:05 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:08:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:08:05 oak-gw06 kernel: Mem-Info: Aug 14 22:08:05 oak-gw06 kernel: active_anon:24810 inactive_anon:51096 isolated_anon:0#012 active_file:389904 inactive_file:2018465 isolated_file:0#012 unevictable:0 dirty:5149 writeback:4515 unstable:0#012 slab_reclaimable:34736 slab_unreclaimable:787625#012 mapped:10282 shmem:45078 pagetables:1695 bounce:0#012 free:631780 free_pcp:1317 free_cma:0 Aug 14 22:08:05 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:08:05 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:08:05 oak-gw06 kernel: Node 0 DMA32 free:543664kB min:69724kB low:87152kB high:104584kB active_anon:16876kB inactive_anon:35588kB active_file:361180kB inactive_file:1421920kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3868kB writeback:720kB mapped:4712kB shmem:31268kB slab_reclaimable:20360kB slab_unreclaimable:415888kB kernel_stack:1008kB pagetables:2104kB unstable:0kB bounce:0kB free_pcp:1932kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:08:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:08:05 oak-gw06 kernel: Node 0 Normal free:1985096kB min:323104kB low:403880kB high:484656kB active_anon:82364kB inactive_anon:168796kB active_file:1198436kB inactive_file:6653112kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:12340kB writeback:17388kB mapped:36416kB shmem:149044kB slab_reclaimable:118584kB slab_unreclaimable:2734596kB kernel_stack:4672kB pagetables:4676kB unstable:0kB bounce:0kB free_pcp:4424kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:08:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:08:05 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:08:05 oak-gw06 kernel: Node 0 DMA32: 5840*4kB (UEM) 6510*8kB (UEM) 8544*16kB (UEM) 6625*32kB (UEM) 1678*64kB (UM) 29*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 535504kB Aug 14 22:08:05 oak-gw06 kernel: Node 0 Normal: 41755*4kB (UEM) 52498*8kB (UEM) 32865*16kB (UEM) 20992*32kB (UEM) 2972*64kB (UEM) 69*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1983628kB Aug 14 22:08:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:08:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:08:05 oak-gw06 kernel: 2090070 total pagecache pages Aug 14 22:08:05 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:08:05 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:08:05 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:08:05 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:08:05 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:08:05 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:08:05 oak-gw06 kernel: 127313 pages reserved Aug 14 22:13:05 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 22:13:05 oak-gw06 kernel: CPU: 6 PID: 4094 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:13:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:13:05 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:13:05 oak-gw06 kernel: 00000000000080d0 0000000081bf1513 ffff88019130b858 ffffffff8168662f Aug 14 22:13:05 oak-gw06 kernel: ffff88019130b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:13:05 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019130b8b8 0000000081bf1513 Aug 14 22:13:05 oak-gw06 kernel: Call Trace: Aug 14 22:13:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:13:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:13:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:13:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:13:05 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 22:13:05 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 22:13:05 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:13:05 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:13:05 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:13:05 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:13:05 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:13:05 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:13:05 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:13:05 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:13:05 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:13:05 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:13:05 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:13:05 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:13:05 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:13:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:13:05 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:13:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:13:05 oak-gw06 kernel: Mem-Info: Aug 14 22:13:05 oak-gw06 kernel: active_anon:19466 inactive_anon:51096 isolated_anon:0#012 active_file:163242 inactive_file:2139425 isolated_file:0#012 unevictable:0 dirty:4131 writeback:1750 unstable:0#012 slab_reclaimable:34707 slab_unreclaimable:784508#012 mapped:10276 shmem:45078 pagetables:1662 bounce:0#012 free:774236 free_pcp:763 free_cma:0 Aug 14 22:13:05 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:13:05 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:13:05 oak-gw06 kernel: Node 0 DMA32 free:750248kB min:69724kB low:87152kB high:104584kB active_anon:11932kB inactive_anon:35588kB active_file:145276kB inactive_file:1453032kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1648kB writeback:2220kB mapped:4716kB shmem:31268kB slab_reclaimable:20332kB slab_unreclaimable:415284kB kernel_stack:976kB pagetables:1364kB unstable:0kB bounce:0kB free_pcp:956kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:13:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:13:05 oak-gw06 kernel: Node 0 Normal free:2384164kB min:323104kB low:403880kB high:484656kB active_anon:65932kB inactive_anon:168796kB active_file:507692kB inactive_file:7058192kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:21084kB writeback:4780kB mapped:36388kB shmem:149044kB slab_reclaimable:118496kB slab_unreclaimable:2722732kB kernel_stack:4736kB pagetables:5284kB unstable:0kB bounce:0kB free_pcp:2936kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:13:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:13:05 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:13:05 oak-gw06 kernel: Node 0 DMA32: 10104*4kB (UEM) 8066*8kB (UEM) 16554*16kB (UEM) 8237*32kB (UEM) 1797*64kB (UM) 75*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 758000kB Aug 14 22:13:05 oak-gw06 kernel: Node 0 Normal: 64656*4kB (UEM) 53671*8kB (UEM) 45770*16kB (UEM) 26255*32kB (UEM) 2369*64kB (UEM) 85*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2422968kB Aug 14 22:13:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:13:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:13:05 oak-gw06 kernel: 2049598 total pagecache pages Aug 14 22:13:05 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:13:05 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:13:05 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:13:05 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:13:05 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:13:05 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:13:05 oak-gw06 kernel: 127313 pages reserved Aug 14 22:13:05 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 22:13:05 oak-gw06 kernel: CPU: 6 PID: 4094 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:13:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:13:05 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:13:05 oak-gw06 kernel: 00000000000080d0 0000000081bf1513 ffff88019130b808 ffffffff8168662f Aug 14 22:13:05 oak-gw06 kernel: ffff88019130b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:13:05 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019130b868 0000000081bf1513 Aug 14 22:13:05 oak-gw06 kernel: Call Trace: Aug 14 22:13:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:13:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:13:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:13:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:13:05 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 22:13:05 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 22:13:05 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 22:13:05 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 22:13:05 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:13:05 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:13:05 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:13:05 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:13:05 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:13:05 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:13:05 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:13:05 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:13:05 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:13:05 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:13:05 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:13:05 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:13:05 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:13:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:13:05 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:13:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:13:05 oak-gw06 kernel: Mem-Info: Aug 14 22:13:05 oak-gw06 kernel: active_anon:19466 inactive_anon:51096 isolated_anon:0#012 active_file:163242 inactive_file:2069801 isolated_file:0#012 unevictable:0 dirty:9563 writeback:1750 unstable:0#012 slab_reclaimable:34707 slab_unreclaimable:784508#012 mapped:10276 shmem:45078 pagetables:1662 bounce:0#012 free:849036 free_pcp:1274 free_cma:0 Aug 14 22:13:05 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:13:05 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:13:05 oak-gw06 kernel: Node 0 DMA32 free:800704kB min:69724kB low:87152kB high:104584kB active_anon:11932kB inactive_anon:35588kB active_file:145276kB inactive_file:1406160kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1648kB writeback:2220kB mapped:4716kB shmem:31268kB slab_reclaimable:20332kB slab_unreclaimable:415284kB kernel_stack:976kB pagetables:1364kB unstable:0kB bounce:0kB free_pcp:2352kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:13:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:13:05 oak-gw06 kernel: Node 0 Normal free:2670724kB min:323104kB low:403880kB high:484656kB active_anon:65932kB inactive_anon:168796kB active_file:507692kB inactive_file:6780512kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:42812kB writeback:4780kB mapped:36388kB shmem:149044kB slab_reclaimable:118496kB slab_unreclaimable:2722732kB kernel_stack:4736kB pagetables:5284kB unstable:0kB bounce:0kB free_pcp:2352kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:13:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:13:05 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:13:05 oak-gw06 kernel: Node 0 DMA32: 13541*4kB (UEM) 14469*8kB (UEM) 16925*16kB (UEM) 8242*32kB (UEM) 1800*64kB (UM) 75*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 829260kB Aug 14 22:13:05 oak-gw06 kernel: Node 0 Normal: 80467*4kB (UEM) 80423*8kB (UEM) 46221*16kB (UEM) 26373*32kB (UEM) 2373*64kB (UEM) 85*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2711476kB Aug 14 22:13:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:13:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:13:05 oak-gw06 kernel: 2055127 total pagecache pages Aug 14 22:13:05 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:13:05 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:13:05 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:13:05 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:13:05 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:13:05 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:13:05 oak-gw06 kernel: 127313 pages reserved Aug 14 22:18:05 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 22:18:05 oak-gw06 kernel: CPU: 6 PID: 4019 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:18:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:18:05 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:18:05 oak-gw06 kernel: 00000000000080d0 00000000cfca4a15 ffff88008be57858 ffffffff8168662f Aug 14 22:18:05 oak-gw06 kernel: ffff88008be578e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:18:05 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88008be578b8 00000000cfca4a15 Aug 14 22:18:05 oak-gw06 kernel: Call Trace: Aug 14 22:18:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:18:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:18:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:18:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:18:05 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 22:18:05 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 22:18:05 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:18:05 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:18:05 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:18:05 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:18:05 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:18:05 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:18:05 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:18:05 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:18:05 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:18:05 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:18:05 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:18:05 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:18:05 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:18:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:18:05 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:18:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:18:05 oak-gw06 kernel: Mem-Info: Aug 14 22:18:05 oak-gw06 kernel: active_anon:24633 inactive_anon:51096 isolated_anon:0#012 active_file:255550 inactive_file:2057121 isolated_file:0#012 unevictable:0 dirty:7828 writeback:3064 unstable:0#012 slab_reclaimable:34644 slab_unreclaimable:775050#012 mapped:10301 shmem:45078 pagetables:1689 bounce:0#012 free:771609 free_pcp:823 free_cma:0 Aug 14 22:18:05 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:18:05 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:18:05 oak-gw06 kernel: Node 0 DMA32 free:786188kB min:69724kB low:87152kB high:104584kB active_anon:11264kB inactive_anon:35588kB active_file:194408kB inactive_file:1380848kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3348kB writeback:1556kB mapped:4708kB shmem:31268kB slab_reclaimable:20316kB slab_unreclaimable:406748kB kernel_stack:976kB pagetables:1372kB unstable:0kB bounce:0kB free_pcp:816kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:18:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:18:05 oak-gw06 kernel: Node 0 Normal free:2284080kB min:323104kB low:403880kB high:484656kB active_anon:87268kB inactive_anon:168796kB active_file:827792kB inactive_file:6856996kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:26072kB writeback:8348kB mapped:36496kB shmem:149044kB slab_reclaimable:118260kB slab_unreclaimable:2693436kB kernel_stack:4704kB pagetables:5384kB unstable:0kB bounce:0kB free_pcp:2060kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:18:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:18:05 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:18:05 oak-gw06 kernel: Node 0 DMA32: 9697*4kB (UEM) 11807*8kB (UEM) 10656*16kB (UEM) 9275*32kB (UEM) 2617*64kB (UM) 147*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 787100kB Aug 14 22:18:05 oak-gw06 kernel: Node 0 Normal: 56973*4kB (UEM) 48138*8kB (UEM) 35044*16kB (UEM) 27409*32kB (UEM) 3389*64kB (UEM) 87*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2278820kB Aug 14 22:18:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:18:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:18:05 oak-gw06 kernel: 2091331 total pagecache pages Aug 14 22:18:05 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:18:05 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:18:05 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:18:05 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:18:05 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:18:05 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:18:05 oak-gw06 kernel: 127313 pages reserved Aug 14 22:18:05 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 22:18:05 oak-gw06 kernel: CPU: 6 PID: 4019 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:18:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:18:05 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:18:05 oak-gw06 kernel: 00000000000080d0 00000000cfca4a15 ffff88008be57808 ffffffff8168662f Aug 14 22:18:05 oak-gw06 kernel: ffff88008be57898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:18:05 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88008be57868 00000000cfca4a15 Aug 14 22:18:05 oak-gw06 kernel: Call Trace: Aug 14 22:18:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:18:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:18:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:18:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:18:05 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 22:18:05 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 22:18:05 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 22:18:05 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 22:18:05 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:18:05 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:18:05 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:18:05 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:18:05 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:18:05 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:18:05 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:18:05 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:18:05 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:18:05 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:18:05 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:18:05 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:18:05 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:18:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:18:05 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:18:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:18:05 oak-gw06 kernel: Mem-Info: Aug 14 22:18:05 oak-gw06 kernel: active_anon:24633 inactive_anon:51096 isolated_anon:0#012 active_file:255550 inactive_file:2056926 isolated_file:0#012 unevictable:0 dirty:2796 writeback:2488 unstable:0#012 slab_reclaimable:34644 slab_unreclaimable:775050#012 mapped:10301 shmem:45078 pagetables:1689 bounce:0#012 free:774715 free_pcp:839 free_cma:0 Aug 14 22:18:05 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:18:05 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:18:05 oak-gw06 kernel: Node 0 DMA32 free:789116kB min:69724kB low:87152kB high:104584kB active_anon:11264kB inactive_anon:35588kB active_file:194408kB inactive_file:1380848kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3348kB writeback:52kB mapped:4708kB shmem:31268kB slab_reclaimable:20316kB slab_unreclaimable:406748kB kernel_stack:976kB pagetables:1372kB unstable:0kB bounce:0kB free_pcp:656kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:18:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:18:05 oak-gw06 kernel: Node 0 Normal free:2273520kB min:323104kB low:403880kB high:484656kB active_anon:87268kB inactive_anon:168796kB active_file:827792kB inactive_file:6865836kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:6284kB writeback:4080kB mapped:36496kB shmem:149044kB slab_reclaimable:118260kB slab_unreclaimable:2693436kB kernel_stack:4704kB pagetables:5384kB unstable:0kB bounce:0kB free_pcp:2100kB local_pcp:4kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:18:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:18:05 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:18:05 oak-gw06 kernel: Node 0 DMA32: 9977*4kB (UEM) 11727*8kB (UEM) 10936*16kB (UEM) 9275*32kB (UEM) 2617*64kB (UM) 147*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 792060kB Aug 14 22:18:05 oak-gw06 kernel: Node 0 Normal: 57299*4kB (UE) 47488*8kB (UEM) 34762*16kB (UEM) 27409*32kB (UEM) 3389*64kB (UEM) 87*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2270412kB Aug 14 22:18:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:18:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:18:05 oak-gw06 kernel: 2088827 total pagecache pages Aug 14 22:18:05 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:18:05 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:18:05 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:18:05 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:18:05 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:18:05 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:18:05 oak-gw06 kernel: 127313 pages reserved Aug 14 22:23:05 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 22:23:05 oak-gw06 kernel: CPU: 6 PID: 4096 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:23:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:23:05 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:23:05 oak-gw06 kernel: 00000000000080d0 000000004508f8e7 ffff8800a17d3858 ffffffff8168662f Aug 14 22:23:05 oak-gw06 kernel: ffff8800a17d38e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:23:05 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a17d38b8 000000004508f8e7 Aug 14 22:23:05 oak-gw06 kernel: Call Trace: Aug 14 22:23:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:23:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:23:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:23:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:23:05 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 22:23:05 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 22:23:05 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:23:05 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:23:05 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:23:05 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:23:05 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:23:05 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:23:05 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:23:05 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:23:05 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:23:05 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:23:05 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:23:05 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:23:05 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:23:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:23:05 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:23:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:23:05 oak-gw06 kernel: Mem-Info: Aug 14 22:23:05 oak-gw06 kernel: active_anon:19969 inactive_anon:51096 isolated_anon:0#012 active_file:318803 inactive_file:2258544 isolated_file:0#012 unevictable:0 dirty:3278 writeback:1826 unstable:0#012 slab_reclaimable:34634 slab_unreclaimable:773676#012 mapped:10302 shmem:45078 pagetables:1667 bounce:0#012 free:512308 free_pcp:789 free_cma:0 Aug 14 22:23:05 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:23:05 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:23:05 oak-gw06 kernel: Node 0 DMA32 free:468504kB min:69724kB low:87152kB high:104584kB active_anon:10896kB inactive_anon:35588kB active_file:264060kB inactive_file:1619532kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1244kB writeback:1376kB mapped:4732kB shmem:31268kB slab_reclaimable:20316kB slab_unreclaimable:411696kB kernel_stack:976kB pagetables:1204kB unstable:0kB bounce:0kB free_pcp:936kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:23:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:23:05 oak-gw06 kernel: Node 0 Normal free:1539824kB min:323104kB low:403880kB high:484656kB active_anon:68980kB inactive_anon:168796kB active_file:1011152kB inactive_file:7447144kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:14584kB writeback:8644kB mapped:36476kB shmem:149044kB slab_reclaimable:118220kB slab_unreclaimable:2682992kB kernel_stack:4720kB pagetables:5464kB unstable:0kB bounce:0kB free_pcp:2552kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:23:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:23:05 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:23:05 oak-gw06 kernel: Node 0 DMA32: 8830*4kB (UE) 8212*8kB (UEM) 880*16kB (UEM) 5036*32kB (UEM) 2399*64kB (UM) 250*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 462296kB Aug 14 22:23:05 oak-gw06 kernel: Node 0 Normal: 57305*4kB (UEM) 47005*8kB (UEM) 8363*16kB (UEM) 17696*32kB (UEM) 3642*64kB (UEM) 116*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1553276kB Aug 14 22:23:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:23:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:23:05 oak-gw06 kernel: 2082210 total pagecache pages Aug 14 22:23:05 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:23:05 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:23:05 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:23:05 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:23:05 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:23:05 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:23:05 oak-gw06 kernel: 127313 pages reserved Aug 14 22:23:05 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 22:23:05 oak-gw06 kernel: CPU: 1 PID: 4096 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:23:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:23:05 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:23:05 oak-gw06 kernel: 00000000000080d0 000000004508f8e7 ffff8800a17d3808 ffffffff8168662f Aug 14 22:23:05 oak-gw06 kernel: ffff8800a17d3898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:23:05 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a17d3868 000000004508f8e7 Aug 14 22:23:05 oak-gw06 kernel: Call Trace: Aug 14 22:23:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:23:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:23:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:23:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:23:05 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 22:23:05 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 22:23:05 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 22:23:05 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 22:23:05 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:23:05 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:23:05 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:23:05 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:23:05 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:23:05 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:23:05 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:23:05 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:23:05 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:23:05 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:23:05 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:23:05 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:23:05 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:23:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:23:05 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:23:05 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:23:05 oak-gw06 kernel: Mem-Info: Aug 14 22:23:05 oak-gw06 kernel: active_anon:20095 inactive_anon:51096 isolated_anon:0#012 active_file:318803 inactive_file:2261616 isolated_file:0#012 unevictable:0 dirty:3757 writeback:1147 unstable:0#012 slab_reclaimable:34634 slab_unreclaimable:773676#012 mapped:10302 shmem:45078 pagetables:1667 bounce:0#012 free:511676 free_pcp:926 free_cma:0 Aug 14 22:23:05 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:23:05 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:23:05 oak-gw06 kernel: Node 0 DMA32 free:456448kB min:69724kB low:87152kB high:104584kB active_anon:11400kB inactive_anon:35588kB active_file:264060kB inactive_file:1625580kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1996kB writeback:1376kB mapped:4732kB shmem:31268kB slab_reclaimable:20316kB slab_unreclaimable:411696kB kernel_stack:976kB pagetables:1204kB unstable:0kB bounce:0kB free_pcp:1028kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:23:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:23:05 oak-gw06 kernel: Node 0 Normal free:1553212kB min:323104kB low:403880kB high:484656kB active_anon:68980kB inactive_anon:168796kB active_file:1011152kB inactive_file:7440124kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:12644kB writeback:1660kB mapped:36476kB shmem:149044kB slab_reclaimable:118220kB slab_unreclaimable:2682992kB kernel_stack:4720kB pagetables:5464kB unstable:0kB bounce:0kB free_pcp:3092kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:23:05 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:23:05 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:23:05 oak-gw06 kernel: Node 0 DMA32: 8980*4kB (UEM) 8238*8kB (UEM) 868*16kB (UEM) 4945*32kB (UEM) 2399*64kB (UM) 250*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 460000kB Aug 14 22:23:05 oak-gw06 kernel: Node 0 Normal: 56883*4kB (UE) 46990*8kB (UE) 8203*16kB (UE) 17600*32kB (UEM) 3642*64kB (UEM) 116*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1545836kB Aug 14 22:23:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:23:05 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:23:05 oak-gw06 kernel: 2076723 total pagecache pages Aug 14 22:23:05 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:23:05 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:23:05 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:23:05 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:23:05 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:23:05 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:23:05 oak-gw06 kernel: 127313 pages reserved Aug 14 22:28:05 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 22:28:05 oak-gw06 kernel: CPU: 7 PID: 4094 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:28:05 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:28:05 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:28:05 oak-gw06 kernel: 00000000000080d0 0000000081bf1513 ffff88019130b858 ffffffff8168662f Aug 14 22:28:05 oak-gw06 kernel: ffff88019130b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:28:05 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019130b8b8 0000000081bf1513 Aug 14 22:28:05 oak-gw06 kernel: Call Trace: Aug 14 22:28:05 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:28:05 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:28:05 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:28:05 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:28:05 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 22:28:05 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 22:28:05 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:28:05 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:28:05 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:28:05 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:28:05 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:28:06 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:28:06 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:28:06 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:28:06 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:28:06 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:28:06 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:28:06 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:28:06 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:28:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:28:06 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:28:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:28:06 oak-gw06 kernel: Mem-Info: Aug 14 22:28:06 oak-gw06 kernel: active_anon:24624 inactive_anon:51096 isolated_anon:0#012 active_file:199549 inactive_file:2280656 isolated_file:0#012 unevictable:0 dirty:2485 writeback:1178 unstable:0#012 slab_reclaimable:34630 slab_unreclaimable:772934#012 mapped:10326 shmem:45078 pagetables:1700 bounce:0#012 free:597096 free_pcp:1191 free_cma:0 Aug 14 22:28:06 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:28:06 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:28:06 oak-gw06 kernel: Node 0 DMA32 free:637704kB min:69724kB low:87152kB high:104584kB active_anon:13796kB inactive_anon:35588kB active_file:169972kB inactive_file:1538144kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:232kB writeback:0kB mapped:4720kB shmem:31268kB slab_reclaimable:20316kB slab_unreclaimable:409824kB kernel_stack:944kB pagetables:1208kB unstable:0kB bounce:0kB free_pcp:980kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:28:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:28:06 oak-gw06 kernel: Node 0 Normal free:1713740kB min:323104kB low:403880kB high:484656kB active_anon:84700kB inactive_anon:168796kB active_file:628224kB inactive_file:7602160kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8156kB writeback:9368kB mapped:36584kB shmem:149044kB slab_reclaimable:118204kB slab_unreclaimable:2681896kB kernel_stack:4752kB pagetables:5592kB unstable:0kB bounce:0kB free_pcp:3172kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:28:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:28:06 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:28:06 oak-gw06 kernel: Node 0 DMA32: 8146*4kB (UEM) 8209*8kB (UEM) 9676*16kB (UEM) 6341*32kB (UEM) 2251*64kB (UM) 258*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 633840kB Aug 14 22:28:06 oak-gw06 kernel: Node 0 Normal: 38640*4kB (UEM) 33689*8kB (UEM) 28170*16kB (UEM) 22731*32kB (UEM) 1867*64kB (UEM) 16*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1723720kB Aug 14 22:28:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:28:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:28:06 oak-gw06 kernel: 2054929 total pagecache pages Aug 14 22:28:06 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:28:06 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:28:06 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:28:06 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:28:06 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:28:06 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:28:06 oak-gw06 kernel: 127313 pages reserved Aug 14 22:28:06 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 22:28:06 oak-gw06 kernel: CPU: 7 PID: 4094 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:28:06 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:28:06 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:28:06 oak-gw06 kernel: 00000000000080d0 0000000081bf1513 ffff88019130b808 ffffffff8168662f Aug 14 22:28:06 oak-gw06 kernel: ffff88019130b898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 22:28:06 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019130b868 0000000081bf1513 Aug 14 22:28:06 oak-gw06 kernel: Call Trace: Aug 14 22:28:06 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:28:06 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:28:06 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 22:28:06 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:28:06 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:28:06 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 22:28:06 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 22:28:06 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 22:28:06 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 22:28:06 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:28:06 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:28:06 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:28:06 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:28:06 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:28:06 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:28:06 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:28:06 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:28:06 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:28:06 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:28:06 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:28:06 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:28:06 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:28:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:28:06 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:28:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:28:06 oak-gw06 kernel: Mem-Info: Aug 14 22:28:06 oak-gw06 kernel: active_anon:24649 inactive_anon:51096 isolated_anon:0#012 active_file:199549 inactive_file:2287879 isolated_file:0#012 unevictable:0 dirty:2195 writeback:784 unstable:0#012 slab_reclaimable:34630 slab_unreclaimable:772934#012 mapped:10326 shmem:45078 pagetables:1700 bounce:0#012 free:585944 free_pcp:912 free_cma:0 Aug 14 22:28:06 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:28:06 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:28:06 oak-gw06 kernel: Node 0 DMA32 free:632356kB min:69724kB low:87152kB high:104584kB active_anon:13896kB inactive_anon:35588kB active_file:169972kB inactive_file:1535316kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:236kB writeback:752kB mapped:4720kB shmem:31268kB slab_reclaimable:20316kB slab_unreclaimable:409824kB kernel_stack:944kB pagetables:1208kB unstable:0kB bounce:0kB free_pcp:644kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:28:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:28:06 oak-gw06 kernel: Node 0 Normal free:1683684kB min:323104kB low:403880kB high:484656kB active_anon:84700kB inactive_anon:168796kB active_file:628224kB inactive_file:7620880kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:6992kB writeback:4324kB mapped:36584kB shmem:149044kB slab_reclaimable:118204kB slab_unreclaimable:2681896kB kernel_stack:4752kB pagetables:5592kB unstable:0kB bounce:0kB free_pcp:3728kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:28:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:28:06 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:28:06 oak-gw06 kernel: Node 0 DMA32: 7504*4kB (UE) 8203*8kB (UE) 9083*16kB (UEM) 6355*32kB (UEM) 2252*64kB (UM) 258*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 622248kB Aug 14 22:28:06 oak-gw06 kernel: Node 0 Normal: 36033*4kB (UEM) 33473*8kB (UEM) 26023*16kB (UEM) 22741*32kB (UEM) 1880*64kB (UEM) 16*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1678364kB Aug 14 22:28:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:28:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:28:06 oak-gw06 kernel: 2067269 total pagecache pages Aug 14 22:28:06 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:28:06 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:28:06 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:28:06 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:28:06 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:28:06 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:28:06 oak-gw06 kernel: 127313 pages reserved Aug 14 22:33:06 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 22:33:06 oak-gw06 kernel: CPU: 6 PID: 4120 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:33:06 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:33:06 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:33:06 oak-gw06 kernel: 00000000000080d0 00000000069adb11 ffff8803c23f3858 ffffffff8168662f Aug 14 22:33:06 oak-gw06 kernel: ffff8803c23f38e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:33:06 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803c23f38b8 00000000069adb11 Aug 14 22:33:06 oak-gw06 kernel: Call Trace: Aug 14 22:33:06 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:33:06 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:33:06 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:33:06 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:33:06 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 22:33:06 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 22:33:06 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:33:06 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:33:06 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:33:06 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:33:06 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:33:06 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:33:06 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:33:06 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:33:06 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:33:06 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:33:06 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:33:06 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:33:06 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:33:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:33:06 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:33:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:33:06 oak-gw06 kernel: Mem-Info: Aug 14 22:33:06 oak-gw06 kernel: active_anon:24819 inactive_anon:51096 isolated_anon:0#012 active_file:393570 inactive_file:2038211 isolated_file:0#012 unevictable:0 dirty:3471 writeback:4575 unstable:0#012 slab_reclaimable:34147 slab_unreclaimable:744774#012 mapped:10337 shmem:45078 pagetables:1700 bounce:0#012 free:686166 free_pcp:481 free_cma:0 Aug 14 22:33:06 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:33:06 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:33:06 oak-gw06 kernel: Node 0 DMA32 free:490644kB min:69724kB low:87152kB high:104584kB active_anon:13796kB inactive_anon:35588kB active_file:326956kB inactive_file:1543560kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1464kB writeback:624kB mapped:4720kB shmem:31268kB slab_reclaimable:19716kB slab_unreclaimable:407668kB kernel_stack:928kB pagetables:1208kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:33:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:33:06 oak-gw06 kernel: Node 0 Normal free:2234544kB min:323104kB low:403880kB high:484656kB active_anon:85480kB inactive_anon:168796kB active_file:1247324kB inactive_file:6612852kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:13928kB writeback:18896kB mapped:36640kB shmem:149044kB slab_reclaimable:116872kB slab_unreclaimable:2571380kB kernel_stack:4784kB pagetables:5592kB unstable:0kB bounce:0kB free_pcp:2468kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:33:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:33:06 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:33:06 oak-gw06 kernel: Node 0 DMA32: 9135*4kB (UEM) 7431*8kB (UEM) 9868*16kB (UEM) 4535*32kB (UEM) 1294*64kB (UEM) 69*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 490900kB Aug 14 22:33:06 oak-gw06 kernel: Node 0 Normal: 50815*4kB (UEM) 53128*8kB (UEM) 53009*16kB (UEM) 20236*32kB (UEM) 1641*64kB (UM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2231436kB Aug 14 22:33:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:33:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:33:06 oak-gw06 kernel: 2063192 total pagecache pages Aug 14 22:33:06 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:33:06 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:33:06 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:33:06 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:33:06 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:33:06 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:33:06 oak-gw06 kernel: 127313 pages reserved Aug 14 22:33:06 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 22:33:06 oak-gw06 kernel: CPU: 1 PID: 4120 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:33:06 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:33:06 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:33:06 oak-gw06 kernel: 00000000000080d0 00000000069adb11 ffff8803c23f3808 ffffffff8168662f Aug 14 22:33:06 oak-gw06 kernel: ffff8803c23f3898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:33:06 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803c23f3868 00000000069adb11 Aug 14 22:33:06 oak-gw06 kernel: Call Trace: Aug 14 22:33:06 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:33:06 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:33:06 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:33:06 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:33:06 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 22:33:06 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 22:33:06 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 22:33:06 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 22:33:06 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:33:06 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:33:06 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:33:06 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:33:06 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:33:06 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:33:06 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:33:06 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:33:06 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:33:06 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:33:06 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:33:06 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:33:06 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:33:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:33:06 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:33:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:33:06 oak-gw06 kernel: Mem-Info: Aug 14 22:33:06 oak-gw06 kernel: active_anon:24819 inactive_anon:51096 isolated_anon:0#012 active_file:393568 inactive_file:2040805 isolated_file:0#012 unevictable:0 dirty:3460 writeback:4026 unstable:0#012 slab_reclaimable:34147 slab_unreclaimable:744766#012 mapped:10340 shmem:45078 pagetables:1700 bounce:0#012 free:684009 free_pcp:290 free_cma:0 Aug 14 22:33:06 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:33:06 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:33:06 oak-gw06 kernel: Node 0 DMA32 free:490900kB min:69724kB low:87152kB high:104584kB active_anon:13796kB inactive_anon:35588kB active_file:326952kB inactive_file:1543732kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1516kB writeback:816kB mapped:4720kB shmem:31268kB slab_reclaimable:19716kB slab_unreclaimable:407668kB kernel_stack:928kB pagetables:1208kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:33:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:33:06 oak-gw06 kernel: Node 0 Normal free:2222840kB min:323104kB low:403880kB high:484656kB active_anon:85740kB inactive_anon:168796kB active_file:1247320kB inactive_file:6623908kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:13876kB writeback:12960kB mapped:36640kB shmem:149044kB slab_reclaimable:116872kB slab_unreclaimable:2571380kB kernel_stack:4784kB pagetables:5592kB unstable:0kB bounce:0kB free_pcp:2172kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:33:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:33:06 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:33:06 oak-gw06 kernel: Node 0 DMA32: 9135*4kB (UEM) 7431*8kB (UEM) 9868*16kB (UEM) 4535*32kB (UEM) 1294*64kB (UEM) 69*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 490900kB Aug 14 22:33:06 oak-gw06 kernel: Node 0 Normal: 50894*4kB (UEM) 51733*8kB (UEM) 53007*16kB (UEM) 20239*32kB (UEM) 1641*64kB (UM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2220656kB Aug 14 22:33:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:33:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:33:06 oak-gw06 kernel: 2065484 total pagecache pages Aug 14 22:33:06 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:33:06 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:33:06 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:33:06 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:33:06 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:33:06 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:33:06 oak-gw06 kernel: 127313 pages reserved Aug 14 22:38:06 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 22:38:06 oak-gw06 kernel: CPU: 7 PID: 4094 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:38:06 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:38:06 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:38:06 oak-gw06 kernel: 00000000000080d0 0000000081bf1513 ffff88019130b858 ffffffff8168662f Aug 14 22:38:06 oak-gw06 kernel: ffff88019130b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:38:06 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019130b8b8 0000000081bf1513 Aug 14 22:38:06 oak-gw06 kernel: Call Trace: Aug 14 22:38:06 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:38:06 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:38:06 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:38:06 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:38:06 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 22:38:06 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 22:38:06 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:38:06 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:38:06 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:38:06 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:38:06 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:38:06 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:38:06 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:38:06 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:38:06 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:38:06 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:38:06 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:38:06 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:38:06 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:38:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:38:06 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:38:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:38:06 oak-gw06 kernel: Mem-Info: Aug 14 22:38:06 oak-gw06 kernel: active_anon:23842 inactive_anon:51096 isolated_anon:0#012 active_file:251780 inactive_file:2291147 isolated_file:0#012 unevictable:0 dirty:8894 writeback:4633 unstable:0#012 slab_reclaimable:34109 slab_unreclaimable:742238#012 mapped:10346 shmem:45078 pagetables:1692 bounce:0#012 free:546672 free_pcp:411 free_cma:0 Aug 14 22:38:06 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:38:06 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:38:06 oak-gw06 kernel: Node 0 DMA32 free:547556kB min:69724kB low:87152kB high:104584kB active_anon:13596kB inactive_anon:35588kB active_file:204028kB inactive_file:1582376kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4472kB writeback:36kB mapped:4720kB shmem:31268kB slab_reclaimable:19672kB slab_unreclaimable:401116kB kernel_stack:928kB pagetables:1208kB unstable:0kB bounce:0kB free_pcp:740kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:38:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:38:06 oak-gw06 kernel: Node 0 Normal free:1597344kB min:323104kB low:403880kB high:484656kB active_anon:82292kB inactive_anon:168796kB active_file:803092kB inactive_file:7606392kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:30328kB writeback:14616kB mapped:36664kB shmem:149044kB slab_reclaimable:116764kB slab_unreclaimable:2567820kB kernel_stack:4752kB pagetables:5560kB unstable:0kB bounce:0kB free_pcp:1152kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:38:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:38:06 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:38:06 oak-gw06 kernel: Node 0 DMA32: 7150*4kB (UEM) 8150*8kB (UEM) 480*16kB (UEM) 6094*32kB (UEM) 3456*64kB (UEM) 259*128kB (UM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 551848kB Aug 14 22:38:06 oak-gw06 kernel: Node 0 Normal: 43482*4kB (UEM) 43593*8kB (UEM) 5487*16kB (UEM) 23004*32kB (UEM) 3932*64kB (UEM) 119*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1613472kB Aug 14 22:38:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:38:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:38:06 oak-gw06 kernel: 2085999 total pagecache pages Aug 14 22:38:06 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:38:06 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:38:06 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:38:06 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:38:06 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:38:06 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:38:06 oak-gw06 kernel: 127313 pages reserved Aug 14 22:38:06 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 22:38:06 oak-gw06 kernel: CPU: 3 PID: 4094 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:38:06 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:38:06 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:38:06 oak-gw06 kernel: 00000000000080d0 0000000081bf1513 ffff88019130b808 ffffffff8168662f Aug 14 22:38:06 oak-gw06 kernel: ffff88019130b898 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Aug 14 22:38:06 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 0000000081bf1513 Aug 14 22:38:06 oak-gw06 kernel: Call Trace: Aug 14 22:38:06 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:38:06 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:38:06 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Aug 14 22:38:06 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:38:06 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:38:06 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 22:38:06 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 22:38:06 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 22:38:06 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 22:38:06 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:38:06 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:38:06 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:38:06 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:38:06 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:38:06 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:38:06 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:38:06 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:38:06 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:38:06 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:38:06 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:38:06 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:38:06 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:38:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:38:06 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:38:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:38:06 oak-gw06 kernel: Mem-Info: Aug 14 22:38:06 oak-gw06 kernel: active_anon:23842 inactive_anon:51096 isolated_anon:0#012 active_file:251780 inactive_file:2289549 isolated_file:0#012 unevictable:0 dirty:8797 writeback:3372 unstable:0#012 slab_reclaimable:34109 slab_unreclaimable:742238#012 mapped:10346 shmem:45078 pagetables:1692 bounce:0#012 free:547583 free_pcp:1265 free_cma:0 Aug 14 22:38:06 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:38:06 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:38:06 oak-gw06 kernel: Node 0 DMA32 free:560196kB min:69724kB low:87152kB high:104584kB active_anon:13596kB inactive_anon:35588kB active_file:204028kB inactive_file:1570784kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4472kB writeback:36kB mapped:4720kB shmem:31268kB slab_reclaimable:19672kB slab_unreclaimable:401116kB kernel_stack:928kB pagetables:1208kB unstable:0kB bounce:0kB free_pcp:2552kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:38:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:38:06 oak-gw06 kernel: Node 0 Normal free:1590032kB min:323104kB low:403880kB high:484656kB active_anon:82032kB inactive_anon:168796kB active_file:803092kB inactive_file:7610292kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:30328kB writeback:12676kB mapped:36664kB shmem:149044kB slab_reclaimable:116764kB slab_unreclaimable:2567820kB kernel_stack:4752kB pagetables:5560kB unstable:0kB bounce:0kB free_pcp:2976kB local_pcp:28kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:38:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:38:06 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:38:06 oak-gw06 kernel: Node 0 DMA32: 7665*4kB (UEM) 8448*8kB (UEM) 740*16kB (UEM) 6143*32kB (UEM) 3456*64kB (UEM) 259*128kB (UM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 562020kB Aug 14 22:38:06 oak-gw06 kernel: Node 0 Normal: 41295*4kB (UEM) 43170*8kB (UE) 5332*16kB (UEM) 22664*32kB (UEM) 3932*64kB (UEM) 119*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1587980kB Aug 14 22:38:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:38:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:38:06 oak-gw06 kernel: 2088496 total pagecache pages Aug 14 22:38:06 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:38:06 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:38:06 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:38:06 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:38:06 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:38:06 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:38:06 oak-gw06 kernel: 127313 pages reserved Aug 14 22:43:06 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 22:43:06 oak-gw06 kernel: CPU: 2 PID: 4094 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:43:06 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:43:06 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:43:06 oak-gw06 kernel: 00000000000080d0 0000000081bf1513 ffff88019130b858 ffffffff8168662f Aug 14 22:43:06 oak-gw06 kernel: ffff88019130b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:43:06 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019130b8b8 0000000081bf1513 Aug 14 22:43:06 oak-gw06 kernel: Call Trace: Aug 14 22:43:06 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:43:06 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:43:06 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:43:06 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:43:06 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 22:43:06 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 22:43:06 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:43:06 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:43:06 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:43:06 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:43:06 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:43:06 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:43:06 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:43:06 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:43:06 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:43:06 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:43:06 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:43:06 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:43:06 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:43:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:43:06 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:43:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:43:06 oak-gw06 kernel: Mem-Info: Aug 14 22:43:06 oak-gw06 kernel: active_anon:24822 inactive_anon:51096 isolated_anon:0#012 active_file:205039 inactive_file:2080191 isolated_file:0#012 unevictable:0 dirty:2566 writeback:3018 unstable:0#012 slab_reclaimable:33872 slab_unreclaimable:736761#012 mapped:10359 shmem:45078 pagetables:1693 bounce:0#012 free:840929 free_pcp:397 free_cma:0 Aug 14 22:43:06 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:43:06 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:43:06 oak-gw06 kernel: Node 0 DMA32 free:784632kB min:69724kB low:87152kB high:104584kB active_anon:13620kB inactive_anon:35588kB active_file:164100kB inactive_file:1413720kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2316kB writeback:240kB mapped:4724kB shmem:31268kB slab_reclaimable:19528kB slab_unreclaimable:401248kB kernel_stack:928kB pagetables:1204kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:43:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:43:06 oak-gw06 kernel: Node 0 Normal free:2556972kB min:323104kB low:403880kB high:484656kB active_anon:85668kB inactive_anon:168796kB active_file:656056kB inactive_file:6911204kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8724kB writeback:6400kB mapped:36712kB shmem:149044kB slab_reclaimable:115960kB slab_unreclaimable:2545780kB kernel_stack:4768kB pagetables:5568kB unstable:0kB bounce:0kB free_pcp:1448kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:43:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:43:06 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:43:06 oak-gw06 kernel: Node 0 DMA32: 13322*4kB (UEM) 16013*8kB (UEM) 16439*16kB (UEM) 7408*32kB (UEM) 1439*64kB (UEM) 121*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 789568kB Aug 14 22:43:06 oak-gw06 kernel: Node 0 Normal: 40397*4kB (UE) 63789*8kB (UEM) 61283*16kB (UEM) 23857*32kB (UEM) 2038*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2551020kB Aug 14 22:43:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:43:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:43:06 oak-gw06 kernel: 2089224 total pagecache pages Aug 14 22:43:06 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:43:06 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:43:06 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:43:06 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:43:06 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:43:06 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:43:06 oak-gw06 kernel: 127313 pages reserved Aug 14 22:43:06 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 22:43:06 oak-gw06 kernel: CPU: 2 PID: 4094 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:43:06 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:43:06 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:43:06 oak-gw06 kernel: 00000000000080d0 0000000081bf1513 ffff88019130b808 ffffffff8168662f Aug 14 22:43:06 oak-gw06 kernel: ffff88019130b898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 22:43:06 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019130b868 0000000081bf1513 Aug 14 22:43:06 oak-gw06 kernel: Call Trace: Aug 14 22:43:06 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:43:06 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:43:06 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 22:43:06 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:43:06 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:43:06 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 22:43:06 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 22:43:06 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 22:43:06 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 22:43:06 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:43:06 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:43:06 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:43:06 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:43:06 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:43:06 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:43:06 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:43:06 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:43:06 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:43:06 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:43:06 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:43:06 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:43:06 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:43:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:43:06 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:43:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:43:06 oak-gw06 kernel: Mem-Info: Aug 14 22:43:06 oak-gw06 kernel: active_anon:24822 inactive_anon:51096 isolated_anon:0#012 active_file:205039 inactive_file:2071297 isolated_file:0#012 unevictable:0 dirty:2857 writeback:1854 unstable:0#012 slab_reclaimable:33872 slab_unreclaimable:736761#012 mapped:10359 shmem:45078 pagetables:1693 bounce:0#012 free:848832 free_pcp:1365 free_cma:0 Aug 14 22:43:06 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:43:06 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:43:06 oak-gw06 kernel: Node 0 DMA32 free:797304kB min:69724kB low:87152kB high:104584kB active_anon:13620kB inactive_anon:35588kB active_file:164100kB inactive_file:1404144kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2316kB writeback:240kB mapped:4724kB shmem:31268kB slab_reclaimable:19528kB slab_unreclaimable:401248kB kernel_stack:928kB pagetables:1204kB unstable:0kB bounce:0kB free_pcp:1936kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:43:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:43:06 oak-gw06 kernel: Node 0 Normal free:2587444kB min:323104kB low:403880kB high:484656kB active_anon:85928kB inactive_anon:168796kB active_file:656056kB inactive_file:6875844kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9112kB writeback:9892kB mapped:36712kB shmem:149044kB slab_reclaimable:115960kB slab_unreclaimable:2545780kB kernel_stack:4768kB pagetables:5568kB unstable:0kB bounce:0kB free_pcp:3408kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:43:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:43:06 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:43:06 oak-gw06 kernel: Node 0 DMA32: 13335*4kB (UEM) 16026*8kB (UEM) 16555*16kB (UEM) 7639*32kB (UEM) 1445*64kB (UEM) 121*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 799356kB Aug 14 22:43:06 oak-gw06 kernel: Node 0 Normal: 41090*4kB (UEM) 64847*8kB (UEM) 61829*16kB (UEM) 24154*32kB (UEM) 2109*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2585040kB Aug 14 22:43:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:43:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:43:06 oak-gw06 kernel: 2073709 total pagecache pages Aug 14 22:43:06 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:43:06 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:43:06 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:43:06 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:43:06 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:43:06 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:43:06 oak-gw06 kernel: 127313 pages reserved Aug 14 22:48:06 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 22:48:06 oak-gw06 kernel: CPU: 6 PID: 4094 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:48:06 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:48:06 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:48:06 oak-gw06 kernel: 00000000000080d0 0000000081bf1513 ffff88019130b858 ffffffff8168662f Aug 14 22:48:06 oak-gw06 kernel: ffff88019130b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:48:06 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019130b8b8 0000000081bf1513 Aug 14 22:48:06 oak-gw06 kernel: Call Trace: Aug 14 22:48:06 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:48:06 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:48:06 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:48:06 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:48:06 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 22:48:06 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 22:48:06 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:48:06 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:48:06 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:48:06 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:48:06 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:48:06 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:48:06 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:48:06 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:48:06 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:48:06 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:48:06 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:48:06 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:48:06 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:48:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:48:06 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:48:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:48:06 oak-gw06 kernel: Mem-Info: Aug 14 22:48:06 oak-gw06 kernel: active_anon:19528 inactive_anon:51096 isolated_anon:0#012 active_file:209910 inactive_file:2292170 isolated_file:0#012 unevictable:0 dirty:3012 writeback:0 unstable:0#012 slab_reclaimable:33866 slab_unreclaimable:738178#012 mapped:10363 shmem:45078 pagetables:1681 bounce:0#012 free:628527 free_pcp:151 free_cma:0 Aug 14 22:48:06 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:48:06 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:48:06 oak-gw06 kernel: Node 0 DMA32 free:681696kB min:69724kB low:87152kB high:104584kB active_anon:11016kB inactive_anon:35588kB active_file:167620kB inactive_file:1514144kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2164kB writeback:0kB mapped:4748kB shmem:31268kB slab_reclaimable:19512kB slab_unreclaimable:403020kB kernel_stack:944kB pagetables:1208kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:48:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:48:06 oak-gw06 kernel: Node 0 Normal free:1805052kB min:323104kB low:403880kB high:484656kB active_anon:67096kB inactive_anon:168796kB active_file:672020kB inactive_file:7665456kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9884kB writeback:0kB mapped:36704kB shmem:149044kB slab_reclaimable:115952kB slab_unreclaimable:2549676kB kernel_stack:4736kB pagetables:5516kB unstable:0kB bounce:0kB free_pcp:1176kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:48:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:48:06 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:48:06 oak-gw06 kernel: Node 0 DMA32: 9260*4kB (UEM) 11251*8kB (UEM) 9080*16kB (UEM) 7752*32kB (UEM) 2286*64kB (UEM) 127*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 683208kB Aug 14 22:48:06 oak-gw06 kernel: Node 0 Normal: 39012*4kB (UEM) 40230*8kB (UEM) 21195*16kB (UEM) 26475*32kB (UEM) 2082*64kB (UEM) 8*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1798480kB Aug 14 22:48:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:48:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:48:06 oak-gw06 kernel: 2073811 total pagecache pages Aug 14 22:48:06 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:48:06 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:48:06 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:48:06 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:48:06 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:48:06 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:48:06 oak-gw06 kernel: 127313 pages reserved Aug 14 22:48:06 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 22:48:06 oak-gw06 kernel: CPU: 6 PID: 4094 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:48:06 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:48:06 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:48:06 oak-gw06 kernel: 00000000000080d0 0000000081bf1513 ffff88019130b808 ffffffff8168662f Aug 14 22:48:06 oak-gw06 kernel: ffff88019130b898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 22:48:06 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88019130b868 0000000081bf1513 Aug 14 22:48:06 oak-gw06 kernel: Call Trace: Aug 14 22:48:06 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:48:06 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:48:06 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 22:48:06 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:48:06 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:48:06 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 22:48:06 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 22:48:06 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 22:48:06 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 22:48:06 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:48:06 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:48:06 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:48:06 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:48:06 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:48:06 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:48:06 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:48:06 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:48:06 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:48:06 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:48:06 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:48:06 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:48:06 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:48:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:48:06 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:48:06 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:48:06 oak-gw06 kernel: Mem-Info: Aug 14 22:48:06 oak-gw06 kernel: active_anon:19528 inactive_anon:51096 isolated_anon:0#012 active_file:209910 inactive_file:2300100 isolated_file:0#012 unevictable:0 dirty:3109 writeback:0 unstable:0#012 slab_reclaimable:33866 slab_unreclaimable:738178#012 mapped:10363 shmem:45078 pagetables:1681 bounce:0#012 free:620655 free_pcp:35 free_cma:0 Aug 14 22:48:06 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:48:06 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:48:06 oak-gw06 kernel: Node 0 DMA32 free:681696kB min:69724kB low:87152kB high:104584kB active_anon:11016kB inactive_anon:35588kB active_file:167620kB inactive_file:1514144kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2164kB writeback:0kB mapped:4748kB shmem:31268kB slab_reclaimable:19512kB slab_unreclaimable:403020kB kernel_stack:944kB pagetables:1208kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:48:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:48:06 oak-gw06 kernel: Node 0 Normal free:1781684kB min:323104kB low:403880kB high:484656kB active_anon:67096kB inactive_anon:168796kB active_file:672020kB inactive_file:7689376kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10272kB writeback:0kB mapped:36704kB shmem:149044kB slab_reclaimable:115952kB slab_unreclaimable:2549676kB kernel_stack:4736kB pagetables:5516kB unstable:0kB bounce:0kB free_pcp:440kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:48:06 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:48:06 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:48:06 oak-gw06 kernel: Node 0 DMA32: 9260*4kB (UEM) 11251*8kB (UEM) 9093*16kB (UEM) 7752*32kB (UEM) 2286*64kB (UEM) 127*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 683416kB Aug 14 22:48:06 oak-gw06 kernel: Node 0 Normal: 39039*4kB (UEM) 38054*8kB (UEM) 21196*16kB (UEM) 26482*32kB (UEM) 2082*64kB (UEM) 8*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1781420kB Aug 14 22:48:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:48:06 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:48:06 oak-gw06 kernel: 2078176 total pagecache pages Aug 14 22:48:06 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:48:06 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:48:06 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:48:06 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:48:06 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:48:06 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:48:06 oak-gw06 kernel: 127313 pages reserved Aug 14 22:53:06 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 22:53:06 oak-gw06 kernel: CPU: 6 PID: 4149 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:53:06 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:53:06 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:53:06 oak-gw06 kernel: 00000000000080d0 0000000027a3c968 ffff8800b9bfb858 ffffffff8168662f Aug 14 22:53:06 oak-gw06 kernel: ffff8800b9bfb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:53:06 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800b9bfb8b8 0000000027a3c968 Aug 14 22:53:06 oak-gw06 kernel: Call Trace: Aug 14 22:53:06 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:53:06 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:53:06 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:53:06 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:53:06 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 22:53:06 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 22:53:06 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:53:06 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:53:06 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:53:06 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:53:06 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:53:07 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:53:07 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:53:07 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:53:07 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:53:07 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:53:07 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:53:07 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:53:07 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:53:07 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:53:07 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:53:07 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:53:07 oak-gw06 kernel: Mem-Info: Aug 14 22:53:07 oak-gw06 kernel: active_anon:24719 inactive_anon:51096 isolated_anon:0#012 active_file:340042 inactive_file:1937604 isolated_file:0#012 unevictable:0 dirty:3172 writeback:515 unstable:0#012 slab_reclaimable:33862 slab_unreclaimable:734898#012 mapped:10383 shmem:45078 pagetables:1704 bounce:0#012 free:848630 free_pcp:726 free_cma:0 Aug 14 22:53:07 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:53:07 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:53:07 oak-gw06 kernel: Node 0 DMA32 free:793896kB min:69724kB low:87152kB high:104584kB active_anon:10984kB inactive_anon:35588kB active_file:264900kB inactive_file:1309364kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2864kB writeback:88kB mapped:4736kB shmem:31268kB slab_reclaimable:19496kB slab_unreclaimable:399012kB kernel_stack:976kB pagetables:1208kB unstable:0kB bounce:0kB free_pcp:208kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:53:07 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:53:07 oak-gw06 kernel: Node 0 Normal free:2584780kB min:323104kB low:403880kB high:484656kB active_anon:87892kB inactive_anon:168796kB active_file:1095268kB inactive_file:6434292kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9048kB writeback:3136kB mapped:36796kB shmem:149044kB slab_reclaimable:115952kB slab_unreclaimable:2540564kB kernel_stack:4736kB pagetables:5608kB unstable:0kB bounce:0kB free_pcp:3088kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:53:07 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:53:07 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:53:07 oak-gw06 kernel: Node 0 DMA32: 10407*4kB (UEM) 7996*8kB (UEM) 12139*16kB (UEM) 7415*32kB (UEM) 3459*64kB (UEM) 299*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 797516kB Aug 14 22:53:07 oak-gw06 kernel: Node 0 Normal: 48768*4kB (UE) 41111*8kB (UEM) 49568*16kB (UEM) 33311*32kB (UEM) 2972*64kB (UEM) 79*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2583320kB Aug 14 22:53:07 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:53:07 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:53:07 oak-gw06 kernel: 2087202 total pagecache pages Aug 14 22:53:07 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:53:07 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:53:07 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:53:07 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:53:07 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:53:07 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:53:07 oak-gw06 kernel: 127313 pages reserved Aug 14 22:53:07 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 22:53:07 oak-gw06 kernel: CPU: 2 PID: 4149 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:53:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:53:07 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:53:07 oak-gw06 kernel: 00000000000080d0 0000000027a3c968 ffff8800b9bfb808 ffffffff8168662f Aug 14 22:53:07 oak-gw06 kernel: ffff8800b9bfb898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:53:07 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800b9bfb868 0000000027a3c968 Aug 14 22:53:07 oak-gw06 kernel: Call Trace: Aug 14 22:53:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:53:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:53:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:53:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:53:07 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 22:53:07 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 22:53:07 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 22:53:07 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 22:53:07 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:53:07 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:53:07 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:53:07 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:53:07 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:53:07 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:53:07 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:53:07 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:53:07 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:53:07 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:53:07 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:53:07 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:53:07 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:53:07 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:53:07 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:53:07 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:53:07 oak-gw06 kernel: Mem-Info: Aug 14 22:53:07 oak-gw06 kernel: active_anon:24760 inactive_anon:51096 isolated_anon:0#012 active_file:339977 inactive_file:1937595 isolated_file:0#012 unevictable:0 dirty:3269 writeback:1247 unstable:0#012 slab_reclaimable:33862 slab_unreclaimable:735086#012 mapped:10388 shmem:45078 pagetables:1704 bounce:0#012 free:845689 free_pcp:1355 free_cma:0 Aug 14 22:53:07 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:53:07 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:53:07 oak-gw06 kernel: Node 0 DMA32 free:809168kB min:69724kB low:87152kB high:104584kB active_anon:10984kB inactive_anon:35588kB active_file:264904kB inactive_file:1295384kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3500kB writeback:476kB mapped:4736kB shmem:31268kB slab_reclaimable:19496kB slab_unreclaimable:399252kB kernel_stack:976kB pagetables:1208kB unstable:0kB bounce:0kB free_pcp:2248kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:53:07 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:53:07 oak-gw06 kernel: Node 0 Normal free:2564452kB min:323104kB low:403880kB high:484656kB active_anon:88184kB inactive_anon:168796kB active_file:1094784kB inactive_file:6454448kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8928kB writeback:4284kB mapped:36824kB shmem:149044kB slab_reclaimable:115952kB slab_unreclaimable:2541524kB kernel_stack:4736kB pagetables:5624kB unstable:0kB bounce:0kB free_pcp:2452kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:53:07 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:53:07 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:53:07 oak-gw06 kernel: Node 0 DMA32: 9237*4kB (UE) 9157*8kB (UEM) 12121*16kB (UEM) 7416*32kB (UEM) 3459*64kB (UEM) 299*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 801868kB Aug 14 22:53:07 oak-gw06 kernel: Node 0 Normal: 49455*4kB (UEM) 40349*8kB (UEM) 48304*16kB (UEM) 33310*32kB (UEM) 2972*64kB (UEM) 79*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2559716kB Aug 14 22:53:07 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:53:07 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:53:07 oak-gw06 kernel: 2092140 total pagecache pages Aug 14 22:53:07 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:53:07 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:53:07 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:53:07 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:53:07 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:53:07 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:53:07 oak-gw06 kernel: 127313 pages reserved Aug 14 22:58:07 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 22:58:07 oak-gw06 kernel: CPU: 6 PID: 4149 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:58:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:58:07 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:58:07 oak-gw06 kernel: 00000000000080d0 0000000027a3c968 ffff8800b9bfb858 ffffffff8168662f Aug 14 22:58:07 oak-gw06 kernel: ffff8800b9bfb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 22:58:07 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800b9bfb8b8 0000000027a3c968 Aug 14 22:58:07 oak-gw06 kernel: Call Trace: Aug 14 22:58:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:58:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:58:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:58:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:58:07 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 22:58:07 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 22:58:07 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:58:07 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:58:07 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:58:07 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:58:07 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:58:07 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:58:07 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:58:07 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:58:07 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:58:07 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:58:07 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:58:07 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:58:07 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:58:07 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:58:07 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:58:07 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:58:07 oak-gw06 kernel: Mem-Info: Aug 14 22:58:07 oak-gw06 kernel: active_anon:20438 inactive_anon:51096 isolated_anon:0#012 active_file:83167 inactive_file:2038739 isolated_file:0#012 unevictable:0 dirty:1714 writeback:2429 unstable:0#012 slab_reclaimable:33817 slab_unreclaimable:728767#012 mapped:10383 shmem:45078 pagetables:1681 bounce:0#012 free:1016737 free_pcp:251 free_cma:0 Aug 14 22:58:07 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:58:07 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:58:07 oak-gw06 kernel: Node 0 DMA32 free:835080kB min:69724kB low:87152kB high:104584kB active_anon:11028kB inactive_anon:35588kB active_file:86364kB inactive_file:1448456kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1500kB writeback:1748kB mapped:4748kB shmem:31268kB slab_reclaimable:19460kB slab_unreclaimable:398764kB kernel_stack:944kB pagetables:1204kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:58:07 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:58:07 oak-gw06 kernel: Node 0 Normal free:3210548kB min:323104kB low:403880kB high:484656kB active_anon:70724kB inactive_anon:168796kB active_file:246304kB inactive_file:6710140kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:4580kB writeback:9908kB mapped:36784kB shmem:149044kB slab_reclaimable:115808kB slab_unreclaimable:2516288kB kernel_stack:4736kB pagetables:5520kB unstable:0kB bounce:0kB free_pcp:2484kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:58:07 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:58:07 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:58:07 oak-gw06 kernel: Node 0 DMA32: 9364*4kB (UEM) 18538*8kB (UEM) 15142*16kB (UEM) 7288*32kB (UEM) 2279*64kB (UEM) 229*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 836672kB Aug 14 22:58:07 oak-gw06 kernel: Node 0 Normal: 48139*4kB (UEM) 95757*8kB (UEM) 67637*16kB (UEM) 29521*32kB (UEM) 3234*64kB (UEM) 97*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3205124kB Aug 14 22:58:07 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:58:07 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:58:07 oak-gw06 kernel: 2028310 total pagecache pages Aug 14 22:58:07 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:58:07 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:58:07 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:58:07 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:58:07 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:58:07 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:58:07 oak-gw06 kernel: 127313 pages reserved Aug 14 22:58:07 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 22:58:07 oak-gw06 kernel: CPU: 6 PID: 4149 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 22:58:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 22:58:07 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 22:58:07 oak-gw06 kernel: 00000000000080d0 0000000027a3c968 ffff8800b9bfb808 ffffffff8168662f Aug 14 22:58:07 oak-gw06 kernel: ffff8800b9bfb898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 22:58:07 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800b9bfb868 0000000027a3c968 Aug 14 22:58:07 oak-gw06 kernel: Call Trace: Aug 14 22:58:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 22:58:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 22:58:07 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 22:58:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 22:58:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 22:58:07 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 22:58:07 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 22:58:07 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 22:58:07 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 22:58:07 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 22:58:07 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 22:58:07 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 22:58:07 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 22:58:07 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 22:58:07 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 22:58:07 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 22:58:07 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 22:58:07 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 22:58:07 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 22:58:07 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 22:58:07 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 22:58:07 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 22:58:07 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:58:07 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 22:58:07 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 22:58:07 oak-gw06 kernel: Mem-Info: Aug 14 22:58:07 oak-gw06 kernel: active_anon:20438 inactive_anon:51096 isolated_anon:0#012 active_file:83167 inactive_file:2043614 isolated_file:0#012 unevictable:0 dirty:1520 writeback:3690 unstable:0#012 slab_reclaimable:33817 slab_unreclaimable:728767#012 mapped:10383 shmem:45078 pagetables:1681 bounce:0#012 free:1011710 free_pcp:392 free_cma:0 Aug 14 22:58:07 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 22:58:07 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 22:58:07 oak-gw06 kernel: Node 0 DMA32 free:837644kB min:69724kB low:87152kB high:104584kB active_anon:11028kB inactive_anon:35588kB active_file:86364kB inactive_file:1448456kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1500kB writeback:1748kB mapped:4748kB shmem:31268kB slab_reclaimable:19460kB slab_unreclaimable:398764kB kernel_stack:944kB pagetables:1204kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:58:07 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 22:58:07 oak-gw06 kernel: Node 0 Normal free:3192056kB min:323104kB low:403880kB high:484656kB active_anon:70724kB inactive_anon:168796kB active_file:246304kB inactive_file:6727040kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:4192kB writeback:14176kB mapped:36784kB shmem:149044kB slab_reclaimable:115808kB slab_unreclaimable:2516288kB kernel_stack:4736kB pagetables:5520kB unstable:0kB bounce:0kB free_pcp:1928kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 22:58:07 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 22:58:07 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 22:58:07 oak-gw06 kernel: Node 0 DMA32: 9364*4kB (UEM) 18538*8kB (UEM) 15368*16kB (UEM) 7288*32kB (UEM) 2279*64kB (UEM) 229*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 840288kB Aug 14 22:58:07 oak-gw06 kernel: Node 0 Normal: 48562*4kB (UEM) 93867*8kB (UEM) 67435*16kB (UEM) 29519*32kB (UEM) 3234*64kB (UEM) 97*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3188400kB Aug 14 22:58:07 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 22:58:07 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 22:58:07 oak-gw06 kernel: 2031608 total pagecache pages Aug 14 22:58:07 oak-gw06 kernel: 16 pages in swap cache Aug 14 22:58:07 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 22:58:07 oak-gw06 kernel: Free swap = 4194036kB Aug 14 22:58:07 oak-gw06 kernel: Total swap = 4194300kB Aug 14 22:58:07 oak-gw06 kernel: 4194203 pages RAM Aug 14 22:58:07 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 22:58:07 oak-gw06 kernel: 127313 pages reserved Aug 14 23:03:07 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 23:03:07 oak-gw06 kernel: CPU: 6 PID: 4120 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:03:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:03:07 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:03:07 oak-gw06 kernel: 00000000000080d0 00000000069adb11 ffff8803c23f3858 ffffffff8168662f Aug 14 23:03:07 oak-gw06 kernel: ffff8803c23f38e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:03:07 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803c23f38b8 00000000069adb11 Aug 14 23:03:07 oak-gw06 kernel: Call Trace: Aug 14 23:03:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:03:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:03:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:03:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:03:07 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 23:03:07 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 23:03:07 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:03:07 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:03:07 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:03:07 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:03:07 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:03:07 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:03:07 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:03:07 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:03:07 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:03:07 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:03:07 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:03:07 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:03:07 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:03:07 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:03:07 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:03:07 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:03:07 oak-gw06 kernel: Mem-Info: Aug 14 23:03:07 oak-gw06 kernel: active_anon:24808 inactive_anon:51096 isolated_anon:0#012 active_file:131466 inactive_file:2377249 isolated_file:0#012 unevictable:0 dirty:4306 writeback:3425 unstable:0#012 slab_reclaimable:33791 slab_unreclaimable:730355#012 mapped:10404 shmem:45078 pagetables:1692 bounce:0#012 free:608439 free_pcp:1665 free_cma:0 Aug 14 23:03:07 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:03:07 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:03:07 oak-gw06 kernel: Node 0 DMA32 free:554656kB min:69724kB low:87152kB high:104584kB active_anon:12936kB inactive_anon:35588kB active_file:134228kB inactive_file:1663988kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1488kB writeback:292kB mapped:4736kB shmem:31268kB slab_reclaimable:19456kB slab_unreclaimable:401212kB kernel_stack:944kB pagetables:1212kB unstable:0kB bounce:0kB free_pcp:2664kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:03:07 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:03:07 oak-gw06 kernel: Node 0 Normal free:1857744kB min:323104kB low:403880kB high:484656kB active_anon:86296kB inactive_anon:168796kB active_file:391636kB inactive_file:7863468kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:26212kB writeback:14572kB mapped:36880kB shmem:149044kB slab_reclaimable:115708kB slab_unreclaimable:2520192kB kernel_stack:4784kB pagetables:5556kB unstable:0kB bounce:0kB free_pcp:4212kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:03:07 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:03:07 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:03:07 oak-gw06 kernel: Node 0 DMA32: 7550*4kB (UEM) 12274*8kB (UEM) 5203*16kB (UEM) 3499*32kB (UEM) 2811*64kB (UEM) 287*128kB (UM) 6*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 541784kB Aug 14 23:03:07 oak-gw06 kernel: Node 0 Normal: 50328*4kB (UEM) 53572*8kB (UEM) 28532*16kB (UEM) 18863*32kB (UEM) 2744*64kB (UEM) 65*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1874208kB Aug 14 23:03:07 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:03:07 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:03:07 oak-gw06 kernel: 2088487 total pagecache pages Aug 14 23:03:07 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:03:07 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:03:07 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:03:07 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:03:07 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:03:07 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:03:07 oak-gw06 kernel: 127313 pages reserved Aug 14 23:03:07 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 23:03:07 oak-gw06 kernel: CPU: 6 PID: 4120 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:03:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:03:07 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:03:07 oak-gw06 kernel: 00000000000080d0 00000000069adb11 ffff8803c23f3808 ffffffff8168662f Aug 14 23:03:07 oak-gw06 kernel: ffff8803c23f3898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 23:03:07 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803c23f3868 00000000069adb11 Aug 14 23:03:07 oak-gw06 kernel: Call Trace: Aug 14 23:03:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:03:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:03:07 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 23:03:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:03:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:03:07 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 23:03:07 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 23:03:07 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 23:03:07 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 23:03:07 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:03:07 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:03:07 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:03:07 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:03:07 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:03:07 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:03:07 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:03:07 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:03:07 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:03:07 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:03:07 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:03:07 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:03:07 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:03:07 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:03:07 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:03:07 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:03:07 oak-gw06 kernel: Mem-Info: Aug 14 23:03:07 oak-gw06 kernel: active_anon:24808 inactive_anon:51096 isolated_anon:0#012 active_file:131466 inactive_file:2386171 isolated_file:0#012 unevictable:0 dirty:4500 writeback:3037 unstable:0#012 slab_reclaimable:33791 slab_unreclaimable:730425#012 mapped:10404 shmem:45078 pagetables:1692 bounce:0#012 free:610220 free_pcp:1792 free_cma:0 Aug 14 23:03:07 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:03:07 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:03:07 oak-gw06 kernel: Node 0 DMA32 free:555976kB min:69724kB low:87152kB high:104584kB active_anon:12936kB inactive_anon:35588kB active_file:134228kB inactive_file:1669532kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1488kB writeback:292kB mapped:4736kB shmem:31268kB slab_reclaimable:19456kB slab_unreclaimable:401212kB kernel_stack:944kB pagetables:1212kB unstable:0kB bounce:0kB free_pcp:3448kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:03:07 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:03:07 oak-gw06 kernel: Node 0 Normal free:1883176kB min:323104kB low:403880kB high:484656kB active_anon:86296kB inactive_anon:168796kB active_file:391636kB inactive_file:7871268kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:12244kB writeback:14960kB mapped:36880kB shmem:149044kB slab_reclaimable:115708kB slab_unreclaimable:2520472kB kernel_stack:4784kB pagetables:5556kB unstable:0kB bounce:0kB free_pcp:3772kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:03:07 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:03:07 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:03:07 oak-gw06 kernel: Node 0 DMA32: 8324*4kB (UEM) 12489*8kB (UEM) 5440*16kB (UEM) 3557*32kB (UEM) 2811*64kB (UEM) 287*128kB (UM) 6*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 552248kB Aug 14 23:03:07 oak-gw06 kernel: Node 0 Normal: 46086*4kB (UE) 56813*8kB (UEM) 29313*16kB (UEM) 18989*32kB (UEM) 2744*64kB (UEM) 65*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1899696kB Aug 14 23:03:07 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:03:07 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:03:07 oak-gw06 kernel: 2084940 total pagecache pages Aug 14 23:03:07 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:03:07 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:03:07 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:03:07 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:03:07 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:03:07 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:03:07 oak-gw06 kernel: 127313 pages reserved Aug 14 23:08:07 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 23:08:07 oak-gw06 kernel: CPU: 6 PID: 4120 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:08:07 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:08:07 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:08:07 oak-gw06 kernel: 00000000000080d0 00000000069adb11 ffff8803c23f3858 ffffffff8168662f Aug 14 23:08:07 oak-gw06 kernel: ffff8803c23f38e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:08:07 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803c23f38b8 00000000069adb11 Aug 14 23:08:07 oak-gw06 kernel: Call Trace: Aug 14 23:08:07 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:08:07 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:08:07 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:08:07 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:08:07 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 23:08:07 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 23:08:07 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:08:07 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:08:07 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:08:07 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:08:07 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:08:07 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:08:07 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:08:07 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:08:07 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:08:07 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:08:07 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:08:07 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:08:07 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:08:07 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:08:07 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:08:07 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:08:07 oak-gw06 kernel: Mem-Info: Aug 14 23:08:07 oak-gw06 kernel: active_anon:24808 inactive_anon:51096 isolated_anon:0#012 active_file:212053 inactive_file:2183383 isolated_file:0#012 unevictable:0 dirty:3306 writeback:1270 unstable:0#012 slab_reclaimable:33791 slab_unreclaimable:729739#012 mapped:10420 shmem:45078 pagetables:1692 bounce:0#012 free:737428 free_pcp:515 free_cma:0 Aug 14 23:08:07 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:08:07 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:08:07 oak-gw06 kernel: Node 0 DMA32 free:695616kB min:69724kB low:87152kB high:104584kB active_anon:12936kB inactive_anon:35588kB active_file:149020kB inactive_file:1519012kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1620kB writeback:432kB mapped:4740kB shmem:31268kB slab_reclaimable:19456kB slab_unreclaimable:400604kB kernel_stack:944kB pagetables:1216kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:08:07 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:08:07 oak-gw06 kernel: Node 0 Normal free:2233560kB min:323104kB low:403880kB high:484656kB active_anon:86296kB inactive_anon:168796kB active_file:699192kB inactive_file:7217900kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10052kB writeback:3872kB mapped:36940kB shmem:149044kB slab_reclaimable:115708kB slab_unreclaimable:2518336kB kernel_stack:4752kB pagetables:5552kB unstable:0kB bounce:0kB free_pcp:2464kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:08:07 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:08:07 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:08:07 oak-gw06 kernel: Node 0 DMA32: 9449*4kB (UEM) 9871*8kB (UEM) 4990*16kB (UEM) 7600*32kB (UEM) 3350*64kB (UEM) 323*128kB (UM) 7*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 697340kB Aug 14 23:08:07 oak-gw06 kernel: Node 0 Normal: 50720*4kB (UEM) 59974*8kB (UEM) 34162*16kB (UEM) 24382*32kB (UEM) 3239*64kB (UEM) 63*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2224848kB Aug 14 23:08:07 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:08:07 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:08:07 oak-gw06 kernel: 2055328 total pagecache pages Aug 14 23:08:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:08:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:08:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:08:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:08:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:08:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:08:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:08:08 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 23:08:08 oak-gw06 kernel: CPU: 6 PID: 4120 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:08:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:08:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:08:08 oak-gw06 kernel: 00000000000080d0 00000000069adb11 ffff8803c23f3808 ffffffff8168662f Aug 14 23:08:08 oak-gw06 kernel: ffff8803c23f3898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 23:08:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803c23f3868 00000000069adb11 Aug 14 23:08:08 oak-gw06 kernel: Call Trace: Aug 14 23:08:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:08:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:08:08 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 23:08:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:08:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:08:08 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 23:08:08 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 23:08:08 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 23:08:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 23:08:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:08:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:08:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:08:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:08:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:08:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:08:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:08:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:08:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:08:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:08:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:08:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:08:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:08:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:08:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:08:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:08:08 oak-gw06 kernel: Mem-Info: Aug 14 23:08:08 oak-gw06 kernel: active_anon:24808 inactive_anon:51096 isolated_anon:0#012 active_file:212053 inactive_file:2191768 isolated_file:0#012 unevictable:0 dirty:2724 writeback:979 unstable:0#012 slab_reclaimable:33791 slab_unreclaimable:729739#012 mapped:10420 shmem:45078 pagetables:1692 bounce:0#012 free:729111 free_pcp:531 free_cma:0 Aug 14 23:08:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:08:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:08:08 oak-gw06 kernel: Node 0 DMA32 free:699272kB min:69724kB low:87152kB high:104584kB active_anon:12936kB inactive_anon:35588kB active_file:149020kB inactive_file:1519012kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1620kB writeback:432kB mapped:4740kB shmem:31268kB slab_reclaimable:19456kB slab_unreclaimable:400604kB kernel_stack:944kB pagetables:1216kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:08:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:08:08 oak-gw06 kernel: Node 0 Normal free:2195080kB min:323104kB low:403880kB high:484656kB active_anon:86296kB inactive_anon:168796kB active_file:699192kB inactive_file:7254040kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10052kB writeback:3096kB mapped:36940kB shmem:149044kB slab_reclaimable:115708kB slab_unreclaimable:2518336kB kernel_stack:4752kB pagetables:5552kB unstable:0kB bounce:0kB free_pcp:2640kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:08:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:08:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:08:08 oak-gw06 kernel: Node 0 DMA32: 9449*4kB (UEM) 9871*8kB (UEM) 5173*16kB (UEM) 7600*32kB (UEM) 3350*64kB (UEM) 323*128kB (UM) 7*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 700268kB Aug 14 23:08:08 oak-gw06 kernel: Node 0 Normal: 50761*4kB (UE) 55248*8kB (UEM) 33971*16kB (UEM) 24382*32kB (UEM) 3239*64kB (UEM) 63*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2184148kB Aug 14 23:08:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:08:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:08:08 oak-gw06 kernel: 2065253 total pagecache pages Aug 14 23:08:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:08:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:08:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:08:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:08:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:08:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:08:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:13:08 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 23:13:08 oak-gw06 kernel: CPU: 6 PID: 4120 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:13:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:13:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:13:08 oak-gw06 kernel: 00000000000080d0 00000000069adb11 ffff8803c23f3858 ffffffff8168662f Aug 14 23:13:08 oak-gw06 kernel: ffff8803c23f38e8 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Aug 14 23:13:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 00000000069adb11 Aug 14 23:13:08 oak-gw06 kernel: Call Trace: Aug 14 23:13:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:13:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:13:08 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Aug 14 23:13:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:13:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:13:08 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 23:13:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 23:13:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:13:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:13:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:13:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:13:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:13:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:13:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:13:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:13:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:13:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:13:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:13:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:13:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:13:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:13:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:13:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:13:08 oak-gw06 kernel: Mem-Info: Aug 14 23:13:08 oak-gw06 kernel: active_anon:23708 inactive_anon:51096 isolated_anon:0#012 active_file:275121 inactive_file:2124459 isolated_file:0#012 unevictable:0 dirty:2908 writeback:62 unstable:0#012 slab_reclaimable:33791 slab_unreclaimable:730075#012 mapped:10433 shmem:45078 pagetables:1689 bounce:0#012 free:734912 free_pcp:71 free_cma:0 Aug 14 23:13:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:13:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:13:08 oak-gw06 kernel: Node 0 DMA32 free:680752kB min:69724kB low:87152kB high:104584kB active_anon:11140kB inactive_anon:35588kB active_file:243488kB inactive_file:1448016kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1100kB writeback:0kB mapped:4752kB shmem:31268kB slab_reclaimable:19456kB slab_unreclaimable:398464kB kernel_stack:944kB pagetables:1204kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:13:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:13:08 oak-gw06 kernel: Node 0 Normal free:2226408kB min:323104kB low:403880kB high:484656kB active_anon:83952kB inactive_anon:168796kB active_file:856996kB inactive_file:7066200kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10532kB writeback:248kB mapped:36980kB shmem:149044kB slab_reclaimable:115708kB slab_unreclaimable:2521820kB kernel_stack:4752kB pagetables:5552kB unstable:0kB bounce:0kB free_pcp:580kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:13:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:13:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:13:08 oak-gw06 kernel: Node 0 DMA32: 9358*4kB (UEM) 7981*8kB (UEM) 10655*16kB (UEM) 5968*32kB (UEM) 2585*64kB (UEM) 404*128kB (UM) 6*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 681424kB Aug 14 23:13:08 oak-gw06 kernel: Node 0 Normal: 48634*4kB (UEM) 42744*8kB (UEM) 52085*16kB (UEM) 20972*32kB (UEM) 2642*64kB (UEM) 46*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2216184kB Aug 14 23:13:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:13:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:13:08 oak-gw06 kernel: 2079690 total pagecache pages Aug 14 23:13:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:13:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:13:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:13:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:13:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:13:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:13:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:13:08 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 23:13:08 oak-gw06 kernel: CPU: 6 PID: 4120 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:13:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:13:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:13:08 oak-gw06 kernel: 00000000000080d0 00000000069adb11 ffff8803c23f3808 ffffffff8168662f Aug 14 23:13:08 oak-gw06 kernel: ffff8803c23f3898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:13:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803c23f3868 00000000069adb11 Aug 14 23:13:08 oak-gw06 kernel: Call Trace: Aug 14 23:13:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:13:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:13:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:13:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:13:08 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 23:13:08 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 23:13:08 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 23:13:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 23:13:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:13:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:13:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:13:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:13:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:13:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:13:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:13:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:13:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:13:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:13:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:13:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:13:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:13:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:13:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:13:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:13:08 oak-gw06 kernel: Mem-Info: Aug 14 23:13:08 oak-gw06 kernel: active_anon:23708 inactive_anon:51096 isolated_anon:0#012 active_file:275121 inactive_file:2143894 isolated_file:0#012 unevictable:0 dirty:2908 writeback:62 unstable:0#012 slab_reclaimable:33791 slab_unreclaimable:730075#012 mapped:10433 shmem:45078 pagetables:1689 bounce:0#012 free:715698 free_pcp:74 free_cma:0 Aug 14 23:13:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:13:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:13:08 oak-gw06 kernel: Node 0 DMA32 free:688816kB min:69724kB low:87152kB high:104584kB active_anon:11140kB inactive_anon:35588kB active_file:243488kB inactive_file:1439448kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1100kB writeback:0kB mapped:4752kB shmem:31268kB slab_reclaimable:19456kB slab_unreclaimable:398464kB kernel_stack:944kB pagetables:1204kB unstable:0kB bounce:0kB free_pcp:708kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:13:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:13:08 oak-gw06 kernel: Node 0 Normal free:2243288kB min:323104kB low:403880kB high:484656kB active_anon:83692kB inactive_anon:168796kB active_file:856996kB inactive_file:7048520kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10532kB writeback:248kB mapped:36980kB shmem:149044kB slab_reclaimable:115708kB slab_unreclaimable:2521820kB kernel_stack:4752kB pagetables:5552kB unstable:0kB bounce:0kB free_pcp:1040kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:13:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:13:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:13:08 oak-gw06 kernel: Node 0 DMA32: 12071*4kB (UEM) 10224*8kB (UEM) 10657*16kB (UEM) 5970*32kB (UEM) 2585*64kB (UEM) 404*128kB (UM) 6*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 710316kB Aug 14 23:13:08 oak-gw06 kernel: Node 0 Normal: 63722*4kB (UEM) 47903*8kB (UEM) 50500*16kB (UEM) 20975*32kB (UEM) 2642*64kB (UEM) 46*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2292544kB Aug 14 23:13:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:13:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:13:08 oak-gw06 kernel: 2094264 total pagecache pages Aug 14 23:13:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:13:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:13:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:13:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:13:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:13:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:13:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:18:08 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 23:18:08 oak-gw06 kernel: CPU: 6 PID: 4120 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:18:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:18:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:18:08 oak-gw06 kernel: 00000000000080d0 00000000069adb11 ffff8803c23f3858 ffffffff8168662f Aug 14 23:18:08 oak-gw06 kernel: ffff8803c23f38e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:18:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803c23f38b8 00000000069adb11 Aug 14 23:18:08 oak-gw06 kernel: Call Trace: Aug 14 23:18:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:18:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:18:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:18:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:18:08 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 23:18:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 23:18:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:18:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:18:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:18:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:18:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:18:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:18:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:18:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:18:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:18:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:18:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:18:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:18:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:18:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:18:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:18:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:18:08 oak-gw06 kernel: Mem-Info: Aug 14 23:18:08 oak-gw06 kernel: active_anon:24592 inactive_anon:51096 isolated_anon:0#012 active_file:368832 inactive_file:2142895 isolated_file:0#012 unevictable:0 dirty:2398 writeback:973 unstable:0#012 slab_reclaimable:33749 slab_unreclaimable:723486#012 mapped:10441 shmem:45078 pagetables:1688 bounce:0#012 free:577537 free_pcp:688 free_cma:0 Aug 14 23:18:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:18:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:18:08 oak-gw06 kernel: Node 0 DMA32 free:617840kB min:69724kB low:87152kB high:104584kB active_anon:11952kB inactive_anon:35588kB active_file:288284kB inactive_file:1440432kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:400kB mapped:4752kB shmem:31268kB slab_reclaimable:19416kB slab_unreclaimable:402852kB kernel_stack:912kB pagetables:1204kB unstable:0kB bounce:0kB free_pcp:656kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:18:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:18:08 oak-gw06 kernel: Node 0 Normal free:1673540kB min:323104kB low:403880kB high:484656kB active_anon:86416kB inactive_anon:168796kB active_file:1187044kB inactive_file:7137908kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10056kB writeback:2716kB mapped:37012kB shmem:149044kB slab_reclaimable:115580kB slab_unreclaimable:2491076kB kernel_stack:4768kB pagetables:5548kB unstable:0kB bounce:0kB free_pcp:1832kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:18:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:18:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:18:08 oak-gw06 kernel: Node 0 DMA32: 7249*4kB (UEM) 8681*8kB (UEM) 8586*16kB (UEM) 6987*32kB (UEM) 2122*64kB (UEM) 184*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 619020kB Aug 14 23:18:08 oak-gw06 kernel: Node 0 Normal: 25543*4kB (UE) 38345*8kB (UEM) 22405*16kB (UEM) 21211*32kB (UEM) 3325*64kB (UEM) 86*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1670228kB Aug 14 23:18:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:18:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:18:08 oak-gw06 kernel: 2087183 total pagecache pages Aug 14 23:18:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:18:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:18:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:18:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:18:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:18:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:18:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:18:08 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 23:18:08 oak-gw06 kernel: CPU: 6 PID: 4120 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:18:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:18:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:18:08 oak-gw06 kernel: 00000000000080d0 00000000069adb11 ffff8803c23f3808 ffffffff8168662f Aug 14 23:18:08 oak-gw06 kernel: ffff8803c23f3898 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Aug 14 23:18:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 00000000069adb11 Aug 14 23:18:08 oak-gw06 kernel: Call Trace: Aug 14 23:18:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:18:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:18:08 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Aug 14 23:18:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:18:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:18:08 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 23:18:08 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 23:18:08 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 23:18:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 23:18:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:18:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:18:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:18:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:18:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:18:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:18:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:18:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:18:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:18:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:18:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:18:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:18:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:18:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:18:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:18:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:18:08 oak-gw06 kernel: Mem-Info: Aug 14 23:18:08 oak-gw06 kernel: active_anon:24722 inactive_anon:51096 isolated_anon:0#012 active_file:368832 inactive_file:2149590 isolated_file:0#012 unevictable:0 dirty:2495 writeback:1555 unstable:0#012 slab_reclaimable:33749 slab_unreclaimable:723690#012 mapped:10441 shmem:45078 pagetables:1688 bounce:0#012 free:570632 free_pcp:1169 free_cma:0 Aug 14 23:18:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:18:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:18:08 oak-gw06 kernel: Node 0 DMA32 free:620376kB min:69724kB low:87152kB high:104584kB active_anon:11952kB inactive_anon:35588kB active_file:288284kB inactive_file:1440432kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:400kB mapped:4752kB shmem:31268kB slab_reclaimable:19416kB slab_unreclaimable:402852kB kernel_stack:912kB pagetables:1204kB unstable:0kB bounce:0kB free_pcp:956kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:18:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:18:08 oak-gw06 kernel: Node 0 Normal free:1639272kB min:323104kB low:403880kB high:484656kB active_anon:86936kB inactive_anon:168796kB active_file:1187044kB inactive_file:7166508kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9668kB writeback:5044kB mapped:37012kB shmem:149044kB slab_reclaimable:115580kB slab_unreclaimable:2491892kB kernel_stack:4768kB pagetables:5548kB unstable:0kB bounce:0kB free_pcp:4036kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:18:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:18:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:18:08 oak-gw06 kernel: Node 0 DMA32: 7444*4kB (UEM) 8682*8kB (UEM) 8829*16kB (UEM) 7000*32kB (UEM) 2122*64kB (UEM) 184*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 624112kB Aug 14 23:18:08 oak-gw06 kernel: Node 0 Normal: 25834*4kB (UEM) 38366*8kB (UEM) 20072*16kB (UEM) 21279*32kB (UEM) 3326*64kB (UEM) 86*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1636472kB Aug 14 23:18:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:18:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:18:08 oak-gw06 kernel: 2089456 total pagecache pages Aug 14 23:18:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:18:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:18:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:18:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:18:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:18:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:18:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:23:08 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 23:23:08 oak-gw06 kernel: CPU: 6 PID: 4226 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:23:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:23:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:23:08 oak-gw06 kernel: 00000000000080d0 00000000c7a44432 ffff8801e4edb858 ffffffff8168662f Aug 14 23:23:08 oak-gw06 kernel: ffff8801e4edb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:23:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801e4edb8b8 00000000c7a44432 Aug 14 23:23:08 oak-gw06 kernel: Call Trace: Aug 14 23:23:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:23:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:23:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:23:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:23:08 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 23:23:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 23:23:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:23:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:23:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:23:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:23:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:23:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:23:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:23:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:23:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:23:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:23:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:23:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:23:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:23:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:23:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:23:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:23:08 oak-gw06 kernel: Mem-Info: Aug 14 23:23:08 oak-gw06 kernel: active_anon:20440 inactive_anon:51096 isolated_anon:0#012 active_file:455616 inactive_file:2498628 isolated_file:0#012 unevictable:0 dirty:3614 writeback:996 unstable:0#012 slab_reclaimable:33284 slab_unreclaimable:706740#012 mapped:10442 shmem:45078 pagetables:1675 bounce:0#012 free:205370 free_pcp:1239 free_cma:0 Aug 14 23:23:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:23:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:23:08 oak-gw06 kernel: Node 0 DMA32 free:281000kB min:69724kB low:87152kB high:104584kB active_anon:15264kB inactive_anon:35588kB active_file:363724kB inactive_file:1709144kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1316kB writeback:76kB mapped:4764kB shmem:31268kB slab_reclaimable:19248kB slab_unreclaimable:404892kB kernel_stack:992kB pagetables:1996kB unstable:0kB bounce:0kB free_pcp:1444kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:23:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:23:08 oak-gw06 kernel: Node 0 Normal free:516632kB min:323104kB low:403880kB high:484656kB active_anon:66496kB inactive_anon:168796kB active_file:1458740kB inactive_file:8293428kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:13528kB writeback:3520kB mapped:37004kB shmem:149044kB slab_reclaimable:113888kB slab_unreclaimable:2422052kB kernel_stack:4736kB pagetables:4704kB unstable:0kB bounce:0kB free_pcp:3916kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:23:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:23:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:23:08 oak-gw06 kernel: Node 0 DMA32: 7060*4kB (UEM) 5600*8kB (UEM) 1333*16kB (UEM) 3183*32kB (UEM) 1322*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 285696kB Aug 14 23:23:08 oak-gw06 kernel: Node 0 Normal: 26896*4kB (UEM) 18941*8kB (UEM) 5372*16kB (UEM) 4546*32kB (UEM) 259*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 507112kB Aug 14 23:23:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:23:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:23:08 oak-gw06 kernel: 2080728 total pagecache pages Aug 14 23:23:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:23:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:23:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:23:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:23:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:23:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:23:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:23:08 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 23:23:08 oak-gw06 kernel: CPU: 6 PID: 4226 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:23:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:23:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:23:08 oak-gw06 kernel: 00000000000080d0 00000000c7a44432 ffff8801e4edb808 ffffffff8168662f Aug 14 23:23:08 oak-gw06 kernel: ffff8801e4edb898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:23:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801e4edb868 00000000c7a44432 Aug 14 23:23:08 oak-gw06 kernel: Call Trace: Aug 14 23:23:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:23:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:23:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:23:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:23:08 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 23:23:08 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 23:23:08 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 23:23:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 23:23:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:23:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:23:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:23:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:23:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:23:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:23:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:23:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:23:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:23:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:23:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:23:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:23:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:23:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:23:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:23:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:23:08 oak-gw06 kernel: Mem-Info: Aug 14 23:23:08 oak-gw06 kernel: active_anon:20440 inactive_anon:51096 isolated_anon:0#012 active_file:455486 inactive_file:2505536 isolated_file:0#012 unevictable:0 dirty:3711 writeback:608 unstable:0#012 slab_reclaimable:33284 slab_unreclaimable:706675#012 mapped:10442 shmem:45078 pagetables:1675 bounce:0#012 free:198067 free_pcp:525 free_cma:0 Aug 14 23:23:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:23:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:23:08 oak-gw06 kernel: Node 0 DMA32 free:287116kB min:69724kB low:87152kB high:104584kB active_anon:15264kB inactive_anon:35588kB active_file:363724kB inactive_file:1709144kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1316kB writeback:76kB mapped:4764kB shmem:31268kB slab_reclaimable:19248kB slab_unreclaimable:404892kB kernel_stack:992kB pagetables:1996kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:23:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:23:08 oak-gw06 kernel: Node 0 Normal free:476644kB min:323104kB low:403880kB high:484656kB active_anon:66496kB inactive_anon:168796kB active_file:1458220kB inactive_file:8322880kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:12752kB writeback:6236kB mapped:37004kB shmem:149044kB slab_reclaimable:113888kB slab_unreclaimable:2421792kB kernel_stack:4736kB pagetables:4704kB unstable:0kB bounce:0kB free_pcp:2900kB local_pcp:4kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:23:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:23:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:23:08 oak-gw06 kernel: Node 0 DMA32: 7415*4kB (UEM) 5602*8kB (UEM) 1474*16kB (UEM) 3183*32kB (UEM) 1322*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 289388kB Aug 14 23:23:08 oak-gw06 kernel: Node 0 Normal: 24971*4kB (UEM) 18035*8kB (UEM) 4822*16kB (UEM) 4280*32kB (UEM) 194*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 470692kB Aug 14 23:23:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:23:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:23:08 oak-gw06 kernel: 2087130 total pagecache pages Aug 14 23:23:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:23:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:23:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:23:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:23:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:23:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:23:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:28:08 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 23:28:08 oak-gw06 kernel: CPU: 6 PID: 4226 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:28:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:28:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:28:08 oak-gw06 kernel: 00000000000080d0 00000000c7a44432 ffff8801e4edb858 ffffffff8168662f Aug 14 23:28:08 oak-gw06 kernel: ffff8801e4edb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:28:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801e4edb8b8 00000000c7a44432 Aug 14 23:28:08 oak-gw06 kernel: Call Trace: Aug 14 23:28:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:28:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:28:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:28:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:28:08 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 23:28:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 23:28:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:28:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:28:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:28:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:28:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:28:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:28:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:28:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:28:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:28:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:28:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:28:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:28:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:28:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:28:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:28:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:28:08 oak-gw06 kernel: Mem-Info: Aug 14 23:28:08 oak-gw06 kernel: active_anon:24598 inactive_anon:51096 isolated_anon:0#012 active_file:343401 inactive_file:2060345 isolated_file:0#012 unevictable:0 dirty:9137 writeback:2980 unstable:0#012 slab_reclaimable:33184 slab_unreclaimable:697529#012 mapped:10478 shmem:45078 pagetables:1682 bounce:0#012 free:725235 free_pcp:139 free_cma:0 Aug 14 23:28:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:28:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:28:08 oak-gw06 kernel: Node 0 DMA32 free:563444kB min:69724kB low:87152kB high:104584kB active_anon:14784kB inactive_anon:35588kB active_file:287472kB inactive_file:1487716kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:10484kB writeback:1548kB mapped:4756kB shmem:31268kB slab_reclaimable:19184kB slab_unreclaimable:395580kB kernel_stack:992kB pagetables:1996kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:28:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:28:08 oak-gw06 kernel: Node 0 Normal free:2318620kB min:323104kB low:403880kB high:484656kB active_anon:83868kB inactive_anon:168796kB active_file:1086132kB inactive_file:6753924kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:25676kB writeback:10372kB mapped:37156kB shmem:149044kB slab_reclaimable:113552kB slab_unreclaimable:2394520kB kernel_stack:4688kB pagetables:4732kB unstable:0kB bounce:0kB free_pcp:436kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:28:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:28:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:28:08 oak-gw06 kernel: Node 0 DMA32: 5585*4kB (UEM) 7440*8kB (UEM) 10537*16kB (UEM) 5444*32kB (UEM) 2003*64kB (UM) 97*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 565524kB Aug 14 23:28:08 oak-gw06 kernel: Node 0 Normal: 29459*4kB (UE) 48790*8kB (UEM) 56185*16kB (UEM) 24325*32kB (UEM) 1974*64kB (UM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2311980kB Aug 14 23:28:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:28:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:28:08 oak-gw06 kernel: 2085274 total pagecache pages Aug 14 23:28:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:28:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:28:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:28:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:28:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:28:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:28:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:28:08 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 23:28:08 oak-gw06 kernel: CPU: 6 PID: 4226 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:28:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:28:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:28:08 oak-gw06 kernel: 00000000000080d0 00000000c7a44432 ffff8801e4edb808 ffffffff8168662f Aug 14 23:28:08 oak-gw06 kernel: ffff8801e4edb898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:28:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801e4edb868 00000000c7a44432 Aug 14 23:28:08 oak-gw06 kernel: Call Trace: Aug 14 23:28:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:28:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:28:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:28:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:28:08 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 23:28:08 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 23:28:08 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 23:28:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 23:28:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:28:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:28:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:28:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:28:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:28:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:28:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:28:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:28:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:28:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:28:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:28:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:28:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:28:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:28:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:28:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:28:08 oak-gw06 kernel: Mem-Info: Aug 14 23:28:08 oak-gw06 kernel: active_anon:24598 inactive_anon:51096 isolated_anon:0#012 active_file:343401 inactive_file:2061515 isolated_file:0#012 unevictable:0 dirty:8943 writeback:2301 unstable:0#012 slab_reclaimable:33184 slab_unreclaimable:697529#012 mapped:10478 shmem:45078 pagetables:1682 bounce:0#012 free:718172 free_pcp:318 free_cma:0 Aug 14 23:28:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:28:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:28:08 oak-gw06 kernel: Node 0 DMA32 free:565992kB min:69724kB low:87152kB high:104584kB active_anon:14784kB inactive_anon:35588kB active_file:287472kB inactive_file:1487716kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:10484kB writeback:1548kB mapped:4756kB shmem:31268kB slab_reclaimable:19184kB slab_unreclaimable:395580kB kernel_stack:992kB pagetables:1996kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:28:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:28:08 oak-gw06 kernel: Node 0 Normal free:2284448kB min:323104kB low:403880kB high:484656kB active_anon:83608kB inactive_anon:168796kB active_file:1086132kB inactive_file:6760944kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:25676kB writeback:8820kB mapped:37156kB shmem:149044kB slab_reclaimable:113552kB slab_unreclaimable:2394520kB kernel_stack:4688kB pagetables:4732kB unstable:0kB bounce:0kB free_pcp:1648kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:28:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:28:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:28:08 oak-gw06 kernel: Node 0 DMA32: 5730*4kB (UEM) 7440*8kB (UEM) 10636*16kB (UEM) 5444*32kB (UEM) 2003*64kB (UM) 97*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 567688kB Aug 14 23:28:08 oak-gw06 kernel: Node 0 Normal: 26966*4kB (UE) 47681*8kB (UEM) 55398*16kB (UEM) 24325*32kB (UEM) 1974*64kB (UM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2280544kB Aug 14 23:28:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:28:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:28:08 oak-gw06 kernel: 2087699 total pagecache pages Aug 14 23:28:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:28:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:28:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:28:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:28:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:28:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:28:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:33:08 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 23:33:08 oak-gw06 kernel: CPU: 6 PID: 4233 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:33:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:33:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:33:08 oak-gw06 kernel: 00000000000080d0 00000000c3fd41e9 ffff88012a2bb858 ffffffff8168662f Aug 14 23:33:08 oak-gw06 kernel: ffff88012a2bb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:33:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88012a2bb8b8 00000000c3fd41e9 Aug 14 23:33:08 oak-gw06 kernel: Call Trace: Aug 14 23:33:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:33:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:33:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:33:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:33:08 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 23:33:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 23:33:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:33:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:33:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:33:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:33:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:33:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:33:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:33:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:33:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:33:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:33:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:33:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:33:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:33:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:33:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:33:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:33:08 oak-gw06 kernel: Mem-Info: Aug 14 23:33:08 oak-gw06 kernel: active_anon:18900 inactive_anon:51096 isolated_anon:0#012 active_file:1608080 inactive_file:433415 isolated_file:0#012 unevictable:0 dirty:24941 writeback:345 unstable:0#012 slab_reclaimable:33184 slab_unreclaimable:694099#012 mapped:10487 shmem:45078 pagetables:1681 bounce:0#012 free:1130477 free_pcp:357 free_cma:0 Aug 14 23:33:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:33:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:33:08 oak-gw06 kernel: Node 0 DMA32 free:1210292kB min:69724kB low:87152kB high:104584kB active_anon:10676kB inactive_anon:35588kB active_file:925268kB inactive_file:253196kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:11268kB writeback:0kB mapped:4756kB shmem:31268kB slab_reclaimable:19184kB slab_unreclaimable:391772kB kernel_stack:944kB pagetables:1196kB unstable:0kB bounce:0kB free_pcp:656kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:33:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:33:08 oak-gw06 kernel: Node 0 Normal free:3289996kB min:323104kB low:403880kB high:484656kB active_anon:65444kB inactive_anon:168796kB active_file:5506792kB inactive_file:1481244kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:88496kB writeback:992kB mapped:37192kB shmem:149044kB slab_reclaimable:113552kB slab_unreclaimable:2384608kB kernel_stack:4816kB pagetables:5528kB unstable:0kB bounce:0kB free_pcp:1804kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:33:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:33:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:33:08 oak-gw06 kernel: Node 0 DMA32: 10136*4kB (UEM) 10899*8kB (UEM) 20828*16kB (UEM) 14171*32kB (UEM) 3983*64kB (UEM) 328*128kB (UM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1212376kB Aug 14 23:33:08 oak-gw06 kernel: Node 0 Normal: 50953*4kB (UEM) 43385*8kB (UEM) 73608*16kB (UEM) 39005*32kB (UEM) 4611*64kB (UEM) 126*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3288012kB Aug 14 23:33:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:33:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:33:08 oak-gw06 kernel: 2087197 total pagecache pages Aug 14 23:33:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:33:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:33:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:33:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:33:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:33:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:33:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:33:09 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 23:33:09 oak-gw06 kernel: CPU: 6 PID: 4233 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:33:09 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:33:09 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:33:09 oak-gw06 kernel: 00000000000080d0 00000000c3fd41e9 ffff88012a2bb808 ffffffff8168662f Aug 14 23:33:09 oak-gw06 kernel: ffff88012a2bb898 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 14 23:33:09 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff88012a2bb898 00000000c3fd41e9 Aug 14 23:33:09 oak-gw06 kernel: Call Trace: Aug 14 23:33:09 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:33:09 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:33:09 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:33:09 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:33:09 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 23:33:09 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 23:33:09 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 23:33:09 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 23:33:09 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:33:09 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:33:09 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:33:09 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:33:09 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:33:09 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:33:09 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:33:09 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:33:09 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:33:09 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:33:09 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:33:09 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:33:09 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:33:09 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:33:09 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:33:09 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:33:09 oak-gw06 kernel: Mem-Info: Aug 14 23:33:09 oak-gw06 kernel: active_anon:19349 inactive_anon:51096 isolated_anon:0#012 active_file:1581231 inactive_file:439937 isolated_file:9#012 unevictable:0 dirty:25520 writeback:1059 unstable:0#012 slab_reclaimable:33184 slab_unreclaimable:694337#012 mapped:10495 shmem:45078 pagetables:1678 bounce:0#012 free:1147048 free_pcp:1729 free_cma:0 Aug 14 23:33:09 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:33:09 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:33:09 oak-gw06 kernel: Node 0 DMA32 free:1209300kB min:69724kB low:87152kB high:104584kB active_anon:10676kB inactive_anon:35588kB active_file:906568kB inactive_file:255736kB unevictable:0kB isolated(anon):0kB isolated(file):36kB present:3129332kB managed:2884592kB mlocked:0kB dirty:11128kB writeback:0kB mapped:4768kB shmem:31268kB slab_reclaimable:19184kB slab_unreclaimable:391836kB kernel_stack:944kB pagetables:1196kB unstable:0kB bounce:0kB free_pcp:4048kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:33:09 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:33:09 oak-gw06 kernel: Node 0 Normal free:3393788kB min:323104kB low:403880kB high:484656kB active_anon:69060kB inactive_anon:168796kB active_file:5403536kB inactive_file:1489312kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:91340kB writeback:6176kB mapped:37212kB shmem:149044kB slab_reclaimable:113552kB slab_unreclaimable:2385224kB kernel_stack:4816kB pagetables:5516kB unstable:0kB bounce:0kB free_pcp:4376kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:33:09 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:33:09 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:33:09 oak-gw06 kernel: Node 0 DMA32: 10764*4kB (UEM) 12459*8kB (UEM) 20400*16kB (UEM) 14110*32kB (UEM) 3954*64kB (UEM) 326*128kB (UM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1216456kB Aug 14 23:33:09 oak-gw06 kernel: Node 0 Normal: 60209*4kB (UEM) 53117*8kB (UEM) 74401*16kB (UEM) 38610*32kB (UEM) 4538*64kB (UEM) 126*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3398268kB Aug 14 23:33:09 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:33:09 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:33:09 oak-gw06 kernel: 2053387 total pagecache pages Aug 14 23:33:09 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:33:09 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:33:09 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:33:09 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:33:09 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:33:09 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:33:09 oak-gw06 kernel: 127313 pages reserved Aug 14 23:38:08 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 23:38:08 oak-gw06 kernel: CPU: 6 PID: 4320 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:38:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:38:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:38:08 oak-gw06 kernel: 00000000000080d0 0000000055b43e91 ffff8802d4d33858 ffffffff8168662f Aug 14 23:38:08 oak-gw06 kernel: ffff8802d4d338e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:38:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8802d4d338b8 0000000055b43e91 Aug 14 23:38:08 oak-gw06 kernel: Call Trace: Aug 14 23:38:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:38:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:38:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:38:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:38:08 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 23:38:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 23:38:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:38:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:38:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:38:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:38:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:38:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:38:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:38:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:38:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:38:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:38:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:38:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:38:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:38:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:38:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:38:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:38:08 oak-gw06 kernel: Mem-Info: Aug 14 23:38:08 oak-gw06 kernel: active_anon:23551 inactive_anon:51096 isolated_anon:0#012 active_file:1696637 inactive_file:352921 isolated_file:0#012 unevictable:0 dirty:10437 writeback:1979 unstable:0#012 slab_reclaimable:35396 slab_unreclaimable:700059#012 mapped:10847 shmem:45078 pagetables:1689 bounce:0#012 free:1112816 free_pcp:409 free_cma:0 Aug 14 23:38:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:38:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:38:08 oak-gw06 kernel: Node 0 DMA32 free:1508824kB min:69724kB low:87152kB high:104584kB active_anon:10936kB inactive_anon:35588kB active_file:679572kB inactive_file:190316kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4600kB writeback:516kB mapped:4892kB shmem:31268kB slab_reclaimable:19240kB slab_unreclaimable:390388kB kernel_stack:944kB pagetables:1200kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:38:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:38:08 oak-gw06 kernel: Node 0 Normal free:2919328kB min:323104kB low:403880kB high:484656kB active_anon:83268kB inactive_anon:168796kB active_file:6106976kB inactive_file:1225528kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:37536kB writeback:6624kB mapped:38496kB shmem:149044kB slab_reclaimable:122344kB slab_unreclaimable:2409832kB kernel_stack:4752kB pagetables:5556kB unstable:0kB bounce:0kB free_pcp:2488kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:38:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:38:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:38:08 oak-gw06 kernel: Node 0 DMA32: 9489*4kB (UEM) 16792*8kB (UEM) 24881*16kB (UEM) 15883*32kB (UEM) 5430*64kB (UEM) 640*128kB (UM) 18*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1512692kB Aug 14 23:38:08 oak-gw06 kernel: Node 0 Normal: 49173*4kB (UEM) 50020*8kB (UEM) 67634*16kB (UEM) 31249*32kB (UEM) 3653*64kB (UEM) 114*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2927348kB Aug 14 23:38:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:38:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:38:08 oak-gw06 kernel: 2087643 total pagecache pages Aug 14 23:38:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:38:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:38:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:38:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:38:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:38:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:38:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:38:08 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 23:38:08 oak-gw06 kernel: CPU: 1 PID: 4320 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:38:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:38:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:38:08 oak-gw06 kernel: 00000000000080d0 0000000055b43e91 ffff8802d4d33808 ffffffff8168662f Aug 14 23:38:08 oak-gw06 kernel: ffff8802d4d33898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:38:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8802d4d33868 0000000055b43e91 Aug 14 23:38:08 oak-gw06 kernel: Call Trace: Aug 14 23:38:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:38:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:38:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:38:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:38:08 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 23:38:08 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 23:38:08 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 23:38:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 23:38:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:38:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:38:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:38:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:38:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:38:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:38:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:38:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:38:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:38:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:38:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:38:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:38:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:38:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:38:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:38:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:38:08 oak-gw06 kernel: Mem-Info: Aug 14 23:38:08 oak-gw06 kernel: active_anon:23551 inactive_anon:51096 isolated_anon:0#012 active_file:1683293 inactive_file:354562 isolated_file:0#012 unevictable:0 dirty:10728 writeback:2464 unstable:0#012 slab_reclaimable:35396 slab_unreclaimable:700059#012 mapped:10847 shmem:45078 pagetables:1689 bounce:0#012 free:1123704 free_pcp:1484 free_cma:0 Aug 14 23:38:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:38:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:38:08 oak-gw06 kernel: Node 0 DMA32 free:1517592kB min:69724kB low:87152kB high:104584kB active_anon:10936kB inactive_anon:35588kB active_file:674028kB inactive_file:187292kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4600kB writeback:516kB mapped:4892kB shmem:31268kB slab_reclaimable:19240kB slab_unreclaimable:390388kB kernel_stack:944kB pagetables:1200kB unstable:0kB bounce:0kB free_pcp:2840kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:38:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:38:08 oak-gw06 kernel: Node 0 Normal free:2970064kB min:323104kB low:403880kB high:484656kB active_anon:84048kB inactive_anon:168796kB active_file:6059144kB inactive_file:1223188kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:37536kB writeback:14384kB mapped:38496kB shmem:149044kB slab_reclaimable:122344kB slab_unreclaimable:2409832kB kernel_stack:4752kB pagetables:5556kB unstable:0kB bounce:0kB free_pcp:2868kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:38:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:38:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:38:08 oak-gw06 kernel: Node 0 DMA32: 10593*4kB (UEM) 17083*8kB (UEM) 24872*16kB (UEM) 15885*32kB (UEM) 5435*64kB (UEM) 640*128kB (UM) 18*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1519676kB Aug 14 23:38:08 oak-gw06 kernel: Node 0 Normal: 55266*4kB (UEM) 52136*8kB (UEM) 67733*16kB (UEM) 31255*32kB (UEM) 3653*64kB (UEM) 114*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2970424kB Aug 14 23:38:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:38:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:38:08 oak-gw06 kernel: 2076723 total pagecache pages Aug 14 23:38:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:38:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:38:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:38:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:38:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:38:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:38:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:43:08 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 23:43:08 oak-gw06 kernel: CPU: 6 PID: 4320 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:43:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:43:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:43:08 oak-gw06 kernel: 00000000000080d0 0000000055b43e91 ffff8802d4d33858 ffffffff8168662f Aug 14 23:43:08 oak-gw06 kernel: ffff8802d4d338e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:43:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8802d4d338b8 0000000055b43e91 Aug 14 23:43:08 oak-gw06 kernel: Call Trace: Aug 14 23:43:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:43:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:43:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:43:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:43:08 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 23:43:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 23:43:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:43:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:43:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:43:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:43:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:43:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:43:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:43:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:43:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:43:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:43:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:43:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:43:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:43:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:43:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:43:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:43:08 oak-gw06 kernel: Mem-Info: Aug 14 23:43:08 oak-gw06 kernel: active_anon:12845 inactive_anon:51096 isolated_anon:0#012 active_file:1836399 inactive_file:192212 isolated_file:0#012 unevictable:0 dirty:6146 writeback:62 unstable:0#012 slab_reclaimable:35443 slab_unreclaimable:703818#012 mapped:10424 shmem:45078 pagetables:1377 bounce:0#012 free:1141716 free_pcp:190 free_cma:0 Aug 14 23:43:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:43:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:43:08 oak-gw06 kernel: Node 0 DMA32 free:1662288kB min:69724kB low:87152kB high:104584kB active_anon:10676kB inactive_anon:35588kB active_file:627684kB inactive_file:111884kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4776kB shmem:31268kB slab_reclaimable:19216kB slab_unreclaimable:389180kB kernel_stack:944kB pagetables:1196kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:43:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:43:08 oak-gw06 kernel: Node 0 Normal free:2888684kB min:323104kB low:403880kB high:484656kB active_anon:41224kB inactive_anon:168796kB active_file:6717912kB inactive_file:656964kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:24584kB writeback:248kB mapped:36920kB shmem:149044kB slab_reclaimable:122556kB slab_unreclaimable:2426076kB kernel_stack:4768kB pagetables:4312kB unstable:0kB bounce:0kB free_pcp:540kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:43:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:43:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:43:08 oak-gw06 kernel: Node 0 DMA32: 29886*4kB (UEM) 29215*8kB (UEM) 20950*16kB (UEM) 15854*32kB (UEM) 5693*64kB (UEM) 756*128kB (UM) 21*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1662288kB Aug 14 23:43:08 oak-gw06 kernel: Node 0 Normal: 54971*4kB (UEM) 51563*8kB (UEM) 71023*16kB (UEM) 27775*32kB (UEM) 3393*64kB (UEM) 114*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2889300kB Aug 14 23:43:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:43:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:43:08 oak-gw06 kernel: 2073673 total pagecache pages Aug 14 23:43:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:43:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:43:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:43:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:43:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:43:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:43:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:43:08 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 14 23:43:08 oak-gw06 kernel: CPU: 6 PID: 4320 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:43:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:43:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:43:08 oak-gw06 kernel: 00000000000080d0 0000000055b43e91 ffff8802d4d33808 ffffffff8168662f Aug 14 23:43:08 oak-gw06 kernel: ffff8802d4d33898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:43:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8802d4d33868 0000000055b43e91 Aug 14 23:43:08 oak-gw06 kernel: Call Trace: Aug 14 23:43:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:43:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:43:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:43:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:43:08 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 23:43:08 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 23:43:08 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 23:43:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 23:43:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:43:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:43:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:43:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:43:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:43:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:43:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:43:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:43:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:43:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:43:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:43:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:43:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:43:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:43:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:43:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:43:08 oak-gw06 kernel: Mem-Info: Aug 14 23:43:08 oak-gw06 kernel: active_anon:12910 inactive_anon:51096 isolated_anon:0#012 active_file:1836334 inactive_file:192212 isolated_file:0#012 unevictable:0 dirty:6146 writeback:62 unstable:0#012 slab_reclaimable:35443 slab_unreclaimable:703818#012 mapped:10424 shmem:45078 pagetables:1377 bounce:0#012 free:1141964 free_pcp:52 free_cma:0 Aug 14 23:43:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:43:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:43:08 oak-gw06 kernel: Node 0 DMA32 free:1662288kB min:69724kB low:87152kB high:104584kB active_anon:10676kB inactive_anon:35588kB active_file:627684kB inactive_file:111884kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4776kB shmem:31268kB slab_reclaimable:19216kB slab_unreclaimable:389180kB kernel_stack:944kB pagetables:1196kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:43:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:43:08 oak-gw06 kernel: Node 0 Normal free:2889676kB min:323104kB low:403880kB high:484656kB active_anon:40964kB inactive_anon:168796kB active_file:6717652kB inactive_file:656964kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:24584kB writeback:248kB mapped:36920kB shmem:149044kB slab_reclaimable:122556kB slab_unreclaimable:2426076kB kernel_stack:4768kB pagetables:4312kB unstable:0kB bounce:0kB free_pcp:324kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:43:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:43:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:43:08 oak-gw06 kernel: Node 0 DMA32: 29886*4kB (UEM) 29215*8kB (UEM) 20950*16kB (UEM) 15854*32kB (UEM) 5693*64kB (UEM) 756*128kB (UM) 21*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1662288kB Aug 14 23:43:08 oak-gw06 kernel: Node 0 Normal: 54926*4kB (UEM) 51629*8kB (UEM) 71040*16kB (UEM) 27776*32kB (UEM) 3393*64kB (UEM) 114*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2889952kB Aug 14 23:43:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:43:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:43:08 oak-gw06 kernel: 2073576 total pagecache pages Aug 14 23:43:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:43:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:43:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:43:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:43:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:43:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:43:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:48:08 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 23:48:08 oak-gw06 kernel: CPU: 6 PID: 4211 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:48:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:48:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:48:08 oak-gw06 kernel: 00000000000080d0 00000000c7450652 ffff8802bef1b858 ffffffff8168662f Aug 14 23:48:08 oak-gw06 kernel: ffff8802bef1b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:48:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8802bef1b8b8 00000000c7450652 Aug 14 23:48:08 oak-gw06 kernel: Call Trace: Aug 14 23:48:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:48:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:48:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:48:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:48:08 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 23:48:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 23:48:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:48:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:48:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:48:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:48:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:48:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:48:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:48:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:48:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:48:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:48:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:48:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:48:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:48:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:48:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:48:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:48:08 oak-gw06 kernel: Mem-Info: Aug 14 23:48:08 oak-gw06 kernel: active_anon:15163 inactive_anon:51096 isolated_anon:0#012 active_file:1588973 inactive_file:359556 isolated_file:0#012 unevictable:0 dirty:4941 writeback:276 unstable:0#012 slab_reclaimable:35630 slab_unreclaimable:706510#012 mapped:10530 shmem:45078 pagetables:1651 bounce:0#012 free:1216018 free_pcp:85 free_cma:0 Aug 14 23:48:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:48:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:48:08 oak-gw06 kernel: Node 0 DMA32 free:1442216kB min:69724kB low:87152kB high:104584kB active_anon:9788kB inactive_anon:35588kB active_file:738276kB inactive_file:222292kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:8kB writeback:0kB mapped:4784kB shmem:31268kB slab_reclaimable:19224kB slab_unreclaimable:389204kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:48:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:48:08 oak-gw06 kernel: Node 0 Normal free:3405232kB min:323104kB low:403880kB high:484656kB active_anon:50864kB inactive_anon:168796kB active_file:5617616kB inactive_file:1215932kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:19756kB writeback:1104kB mapped:37336kB shmem:149044kB slab_reclaimable:123296kB slab_unreclaimable:2436820kB kernel_stack:4784kB pagetables:5544kB unstable:0kB bounce:0kB free_pcp:912kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:48:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:48:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:48:08 oak-gw06 kernel: Node 0 DMA32: 29910*4kB (UEM) 20034*8kB (UEM) 9401*16kB (UEM) 13454*32kB (UEM) 6596*64kB (UEM) 1132*128kB (UM) 56*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1442232kB Aug 14 23:48:08 oak-gw06 kernel: Node 0 Normal: 83290*4kB (UEM) 98561*8kB (UEM) 71228*16kB (UEM) 28242*32kB (UEM) 3513*64kB (UEM) 121*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3405360kB Aug 14 23:48:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:48:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:48:08 oak-gw06 kernel: 1993603 total pagecache pages Aug 14 23:48:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:48:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:48:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:48:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:48:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:48:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:48:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:48:08 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 14 23:48:08 oak-gw06 kernel: CPU: 6 PID: 4211 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:48:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:48:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:48:08 oak-gw06 kernel: 00000000000080d0 00000000c7450652 ffff8802bef1b808 ffffffff8168662f Aug 14 23:48:08 oak-gw06 kernel: ffff8802bef1b898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 14 23:48:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8802bef1b868 00000000c7450652 Aug 14 23:48:08 oak-gw06 kernel: Call Trace: Aug 14 23:48:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:48:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:48:08 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 14 23:48:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:48:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:48:08 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 23:48:08 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 23:48:08 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 23:48:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 23:48:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:48:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:48:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:48:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:48:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:48:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:48:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:48:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:48:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:48:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:48:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:48:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:48:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:48:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:48:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:48:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:48:08 oak-gw06 kernel: Mem-Info: Aug 14 23:48:08 oak-gw06 kernel: active_anon:15293 inactive_anon:51096 isolated_anon:0#012 active_file:1588908 inactive_file:359556 isolated_file:0#012 unevictable:0 dirty:4941 writeback:179 unstable:0#012 slab_reclaimable:35630 slab_unreclaimable:706510#012 mapped:10530 shmem:45078 pagetables:1651 bounce:0#012 free:1215958 free_pcp:273 free_cma:0 Aug 14 23:48:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:48:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:48:08 oak-gw06 kernel: Node 0 DMA32 free:1442216kB min:69724kB low:87152kB high:104584kB active_anon:9788kB inactive_anon:35588kB active_file:738276kB inactive_file:222292kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:8kB writeback:0kB mapped:4784kB shmem:31268kB slab_reclaimable:19224kB slab_unreclaimable:389204kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:48:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:48:08 oak-gw06 kernel: Node 0 Normal free:3401536kB min:323104kB low:403880kB high:484656kB active_anon:55284kB inactive_anon:168796kB active_file:5617356kB inactive_file:1215932kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:19756kB writeback:716kB mapped:37336kB shmem:149044kB slab_reclaimable:123296kB slab_unreclaimable:2436820kB kernel_stack:4784kB pagetables:5544kB unstable:0kB bounce:0kB free_pcp:1144kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:48:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:48:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:48:08 oak-gw06 kernel: Node 0 DMA32: 29910*4kB (UEM) 20034*8kB (UEM) 9401*16kB (UEM) 13454*32kB (UEM) 6596*64kB (UEM) 1132*128kB (UM) 56*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1442232kB Aug 14 23:48:08 oak-gw06 kernel: Node 0 Normal: 82216*4kB (UEM) 98561*8kB (UEM) 71220*16kB (UEM) 28242*32kB (UEM) 3513*64kB (UEM) 121*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3400936kB Aug 14 23:48:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:48:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:48:08 oak-gw06 kernel: 1993506 total pagecache pages Aug 14 23:48:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:48:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:48:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:48:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:48:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:48:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:48:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:53:08 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 23:53:08 oak-gw06 kernel: CPU: 6 PID: 4557 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:53:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:53:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:53:08 oak-gw06 kernel: 00000000000080d0 0000000009796b3b ffff88016900b858 ffffffff8168662f Aug 14 23:53:08 oak-gw06 kernel: ffff88016900b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:53:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88016900b8b8 0000000009796b3b Aug 14 23:53:08 oak-gw06 kernel: Call Trace: Aug 14 23:53:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:53:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:53:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:53:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:53:08 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 14 23:53:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 14 23:53:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:53:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:53:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:53:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:53:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:53:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:53:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:53:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:53:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:53:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:53:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:53:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:53:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:53:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:53:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:53:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:53:08 oak-gw06 kernel: Mem-Info: Aug 14 23:53:08 oak-gw06 kernel: active_anon:16547 inactive_anon:51096 isolated_anon:0#012 active_file:1853279 inactive_file:200346 isolated_file:0#012 unevictable:0 dirty:5644 writeback:485 unstable:0#012 slab_reclaimable:36787 slab_unreclaimable:712060#012 mapped:10534 shmem:45078 pagetables:1648 bounce:0#012 free:1102741 free_pcp:167 free_cma:0 Aug 14 23:53:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:53:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:53:08 oak-gw06 kernel: Node 0 DMA32 free:1785544kB min:69724kB low:87152kB high:104584kB active_anon:9788kB inactive_anon:35588kB active_file:502856kB inactive_file:115664kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:28kB writeback:0kB mapped:4780kB shmem:31268kB slab_reclaimable:19220kB slab_unreclaimable:387928kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:53:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:53:08 oak-gw06 kernel: Node 0 Normal free:2605916kB min:323104kB low:403880kB high:484656kB active_anon:57700kB inactive_anon:168796kB active_file:6911040kB inactive_file:685200kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:22548kB writeback:1940kB mapped:37356kB shmem:149044kB slab_reclaimable:127928kB slab_unreclaimable:2460296kB kernel_stack:4816kB pagetables:5532kB unstable:0kB bounce:0kB free_pcp:460kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:53:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:53:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:53:08 oak-gw06 kernel: Node 0 DMA32: 32696*4kB (UEM) 30647*8kB (UEM) 18312*16kB (UEM) 14920*32kB (UEM) 6931*64kB (UEM) 1350*128kB (UM) 87*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1785560kB Aug 14 23:53:08 oak-gw06 kernel: Node 0 Normal: 43631*4kB (UEM) 55841*8kB (UEM) 60217*16kB (UEM) 24787*32kB (UEM) 3360*64kB (UEM) 122*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2608564kB Aug 14 23:53:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:53:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:53:08 oak-gw06 kernel: 2098955 total pagecache pages Aug 14 23:53:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:53:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:53:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:53:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:53:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:53:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:53:08 oak-gw06 kernel: 127313 pages reserved Aug 14 23:53:08 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 14 23:53:08 oak-gw06 kernel: CPU: 6 PID: 4557 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 14 23:53:08 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 14 23:53:08 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 14 23:53:08 oak-gw06 kernel: 00000000000080d0 0000000009796b3b ffff88016900b808 ffffffff8168662f Aug 14 23:53:08 oak-gw06 kernel: ffff88016900b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 14 23:53:08 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88016900b868 0000000009796b3b Aug 14 23:53:08 oak-gw06 kernel: Call Trace: Aug 14 23:53:08 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 14 23:53:08 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 14 23:53:08 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 14 23:53:08 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 14 23:53:08 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 14 23:53:08 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 14 23:53:08 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 14 23:53:08 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 14 23:53:08 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 14 23:53:08 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 14 23:53:08 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 14 23:53:08 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 14 23:53:08 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 14 23:53:08 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 14 23:53:08 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 14 23:53:08 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 14 23:53:08 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 14 23:53:08 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 14 23:53:08 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 14 23:53:08 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 14 23:53:08 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 14 23:53:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:53:08 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 14 23:53:08 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 14 23:53:08 oak-gw06 kernel: Mem-Info: Aug 14 23:53:08 oak-gw06 kernel: active_anon:16807 inactive_anon:51096 isolated_anon:0#012 active_file:1853669 inactive_file:200281 isolated_file:0#012 unevictable:0 dirty:5741 writeback:485 unstable:0#012 slab_reclaimable:36787 slab_unreclaimable:712060#012 mapped:10534 shmem:45078 pagetables:1648 bounce:0#012 free:1101662 free_pcp:127 free_cma:0 Aug 14 23:53:08 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 14 23:53:08 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 14 23:53:08 oak-gw06 kernel: Node 0 DMA32 free:1785544kB min:69724kB low:87152kB high:104584kB active_anon:9788kB inactive_anon:35588kB active_file:502856kB inactive_file:115664kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:28kB writeback:0kB mapped:4780kB shmem:31268kB slab_reclaimable:19220kB slab_unreclaimable:387928kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:53:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 14 23:53:08 oak-gw06 kernel: Node 0 Normal free:2605972kB min:323104kB low:403880kB high:484656kB active_anon:54060kB inactive_anon:168796kB active_file:6912860kB inactive_file:685460kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:22160kB writeback:3104kB mapped:37356kB shmem:149044kB slab_reclaimable:127928kB slab_unreclaimable:2460568kB kernel_stack:4816kB pagetables:5532kB unstable:0kB bounce:0kB free_pcp:724kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 14 23:53:08 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 14 23:53:08 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 14 23:53:08 oak-gw06 kernel: Node 0 DMA32: 32696*4kB (UEM) 30647*8kB (UEM) 18312*16kB (UEM) 14920*32kB (UEM) 6931*64kB (UEM) 1350*128kB (UM) 87*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1785560kB Aug 14 23:53:08 oak-gw06 kernel: Node 0 Normal: 43375*4kB (UEM) 55755*8kB (UEM) 60075*16kB (UEM) 24786*32kB (UEM) 3360*64kB (UEM) 122*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2604548kB Aug 14 23:53:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 14 23:53:08 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 14 23:53:08 oak-gw06 kernel: 2099149 total pagecache pages Aug 14 23:53:08 oak-gw06 kernel: 16 pages in swap cache Aug 14 23:53:08 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 14 23:53:08 oak-gw06 kernel: Free swap = 4194036kB Aug 14 23:53:08 oak-gw06 kernel: Total swap = 4194300kB Aug 14 23:53:08 oak-gw06 kernel: 4194203 pages RAM Aug 14 23:53:08 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 14 23:53:08 oak-gw06 kernel: 127313 pages reserved Aug 15 00:48:09 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 00:48:09 oak-gw06 kernel: CPU: 4 PID: 4979 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 00:48:09 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 00:48:09 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 00:48:09 oak-gw06 kernel: 00000000000080d0 00000000cc1f8d7a ffff88035fbaf858 ffffffff8168662f Aug 15 00:48:09 oak-gw06 kernel: ffff88035fbaf8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 00:48:09 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88035fbaf8b8 00000000cc1f8d7a Aug 15 00:48:09 oak-gw06 kernel: Call Trace: Aug 15 00:48:09 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 00:48:09 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 00:48:09 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 00:48:09 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 00:48:09 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 00:48:09 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 00:48:09 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 00:48:09 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 00:48:09 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 00:48:09 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 00:48:09 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 00:48:09 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 00:48:09 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 00:48:09 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 00:48:09 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 00:48:09 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 00:48:09 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 00:48:09 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 00:48:09 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 00:48:09 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 00:48:09 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 00:48:09 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 00:48:09 oak-gw06 kernel: Mem-Info: Aug 15 00:48:09 oak-gw06 kernel: active_anon:24600 inactive_anon:51096 isolated_anon:0#012 active_file:370488 inactive_file:2009888 isolated_file:0#012 unevictable:0 dirty:2667 writeback:964 unstable:0#012 slab_reclaimable:35429 slab_unreclaimable:723217#012 mapped:10562 shmem:45078 pagetables:1691 bounce:0#012 free:675463 free_pcp:407 free_cma:0 Aug 15 00:48:09 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 00:48:09 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 00:48:09 oak-gw06 kernel: Node 0 DMA32 free:623864kB min:69724kB low:87152kB high:104584kB active_anon:13168kB inactive_anon:35588kB active_file:287568kB inactive_file:1410632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1876kB writeback:0kB mapped:4776kB shmem:31268kB slab_reclaimable:19036kB slab_unreclaimable:385504kB kernel_stack:944kB pagetables:1080kB unstable:0kB bounce:0kB free_pcp:620kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 00:48:09 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 00:48:09 oak-gw06 kernel: Node 0 Normal free:2027568kB min:323104kB low:403880kB high:484656kB active_anon:85752kB inactive_anon:168796kB active_file:1202704kB inactive_file:6640880kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:10732kB writeback:1960kB mapped:37472kB shmem:149044kB slab_reclaimable:122680kB slab_unreclaimable:2509252kB kernel_stack:4752kB pagetables:5684kB unstable:0kB bounce:0kB free_pcp:992kB local_pcp:4kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 00:48:09 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 00:48:09 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 00:48:09 oak-gw06 kernel: Node 0 DMA32: 1572*4kB (UEM) 7177*8kB (UEM) 5136*16kB (UEM) 6566*32kB (UEM) 1933*64kB (UEM) 968*128kB (UM) 82*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 625112kB Aug 15 00:48:09 oak-gw06 kernel: Node 0 Normal: 6094*4kB (UE) 79591*8kB (UEM) 45400*16kB (EM) 13551*32kB (UEM) 3022*64kB (UEM) 48*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2020688kB Aug 15 00:48:09 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 00:48:09 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 00:48:09 oak-gw06 kernel: 2102445 total pagecache pages Aug 15 00:48:09 oak-gw06 kernel: 16 pages in swap cache Aug 15 00:48:09 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 00:48:09 oak-gw06 kernel: Free swap = 4194036kB Aug 15 00:48:09 oak-gw06 kernel: Total swap = 4194300kB Aug 15 00:48:09 oak-gw06 kernel: 4194203 pages RAM Aug 15 00:48:09 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 00:48:09 oak-gw06 kernel: 127313 pages reserved Aug 15 00:48:09 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 00:48:09 oak-gw06 kernel: CPU: 4 PID: 4979 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 00:48:09 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 00:48:09 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 00:48:09 oak-gw06 kernel: 00000000000080d0 00000000cc1f8d7a ffff88035fbaf808 ffffffff8168662f Aug 15 00:48:09 oak-gw06 kernel: ffff88035fbaf898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 00:48:09 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88035fbaf868 00000000cc1f8d7a Aug 15 00:48:09 oak-gw06 kernel: Call Trace: Aug 15 00:48:09 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 00:48:09 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 00:48:09 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 00:48:09 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 00:48:09 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 00:48:09 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 00:48:09 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 00:48:09 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 00:48:09 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 00:48:09 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 00:48:09 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 00:48:09 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 00:48:09 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 00:48:09 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 00:48:09 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 00:48:09 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 00:48:09 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 00:48:09 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 00:48:09 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 00:48:09 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 00:48:09 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 00:48:09 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 00:48:09 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 00:48:09 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 00:48:09 oak-gw06 kernel: Mem-Info: Aug 15 00:48:09 oak-gw06 kernel: active_anon:24600 inactive_anon:51096 isolated_anon:0#012 active_file:378028 inactive_file:2007398 isolated_file:0#012 unevictable:0 dirty:2934 writeback:479 unstable:0#012 slab_reclaimable:35429 slab_unreclaimable:723833#012 mapped:10562 shmem:45078 pagetables:1691 bounce:0#012 free:662447 free_pcp:1189 free_cma:0 Aug 15 00:48:09 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 00:48:09 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 00:48:09 oak-gw06 kernel: Node 0 DMA32 free:617580kB min:69724kB low:87152kB high:104584kB active_anon:13168kB inactive_anon:35588kB active_file:291600kB inactive_file:1411164kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2168kB writeback:732kB mapped:4776kB shmem:31268kB slab_reclaimable:19036kB slab_unreclaimable:386336kB kernel_stack:944kB pagetables:1080kB unstable:0kB bounce:0kB free_pcp:2532kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 00:48:09 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 00:48:09 oak-gw06 kernel: Node 0 Normal free:2022320kB min:323104kB low:403880kB high:484656kB active_anon:85232kB inactive_anon:168796kB active_file:1230004kB inactive_file:6599540kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9180kB writeback:1960kB mapped:37472kB shmem:149044kB slab_reclaimable:122680kB slab_unreclaimable:2508980kB kernel_stack:4752kB pagetables:5684kB unstable:0kB bounce:0kB free_pcp:3300kB local_pcp:16kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 00:48:09 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 00:48:09 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 00:48:09 oak-gw06 kernel: Node 0 DMA32: 1753*4kB (UEM) 7939*8kB (UEM) 4324*16kB (UEM) 6550*32kB (UEM) 1933*64kB (UEM) 968*128kB (UM) 82*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 618428kB Aug 15 00:48:09 oak-gw06 kernel: Node 0 Normal: 6583*4kB (UEM) 81269*8kB (UEM) 45403*16kB (UEM) 13325*32kB (UEM) 3022*64kB (UEM) 48*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2028884kB Aug 15 00:48:09 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 00:48:09 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 00:48:09 oak-gw06 kernel: 2098606 total pagecache pages Aug 15 00:48:09 oak-gw06 kernel: 16 pages in swap cache Aug 15 00:48:09 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 00:48:09 oak-gw06 kernel: Free swap = 4194036kB Aug 15 00:48:09 oak-gw06 kernel: Total swap = 4194300kB Aug 15 00:48:09 oak-gw06 kernel: 4194203 pages RAM Aug 15 00:48:09 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 00:48:09 oak-gw06 kernel: 127313 pages reserved Aug 15 00:53:09 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 15 00:53:09 oak-gw06 kernel: CPU: 6 PID: 5238 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 00:53:09 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 00:53:09 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 00:53:09 oak-gw06 kernel: 00000000000080d0 0000000001751c7d ffff8803f826b858 ffffffff8168662f Aug 15 00:53:09 oak-gw06 kernel: ffff8803f826b8e8 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Aug 15 00:53:09 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 0000000001751c7d Aug 15 00:53:09 oak-gw06 kernel: Call Trace: Aug 15 00:53:09 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 00:53:09 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 00:53:09 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Aug 15 00:53:09 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 00:53:09 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 00:53:09 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 00:53:09 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 00:53:09 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 00:53:09 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 00:53:09 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 00:53:09 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 00:53:09 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 00:53:09 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 00:53:09 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 00:53:09 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 00:53:09 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 00:53:09 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 00:53:09 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 00:53:09 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 00:53:09 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 00:53:09 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 00:53:09 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 00:53:09 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 00:53:09 oak-gw06 kernel: Mem-Info: Aug 15 00:53:09 oak-gw06 kernel: active_anon:23940 inactive_anon:51096 isolated_anon:0#012 active_file:538862 inactive_file:2242716 isolated_file:0#012 unevictable:0 dirty:15101 writeback:4476 unstable:0#012 slab_reclaimable:34017 slab_unreclaimable:715038#012 mapped:10577 shmem:45078 pagetables:1679 bounce:0#012 free:331044 free_pcp:223 free_cma:0 Aug 15 00:53:09 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 00:53:09 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 00:53:09 oak-gw06 kernel: Node 0 DMA32 free:438300kB min:69724kB low:87152kB high:104584kB active_anon:13252kB inactive_anon:35588kB active_file:397828kB inactive_file:1511272kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:11192kB writeback:2840kB mapped:4792kB shmem:31268kB slab_reclaimable:18736kB slab_unreclaimable:386188kB kernel_stack:944kB pagetables:1076kB unstable:0kB bounce:0kB free_pcp:728kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 00:53:09 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 00:53:09 oak-gw06 kernel: Node 0 Normal free:883536kB min:323104kB low:403880kB high:484656kB active_anon:82508kB inactive_anon:168796kB active_file:1757620kB inactive_file:7434420kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:47272kB writeback:16228kB mapped:37516kB shmem:149044kB slab_reclaimable:117332kB slab_unreclaimable:2474484kB kernel_stack:4752kB pagetables:5640kB unstable:0kB bounce:0kB free_pcp:2404kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 00:53:09 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 00:53:09 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 00:53:09 oak-gw06 kernel: Node 0 DMA32: 3326*4kB (UEM) 5934*8kB (UEM) 483*16kB (UEM) 5992*32kB (UEM) 2058*64kB (UM) 270*128kB (UM) 32*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 434712kB Aug 15 00:53:09 oak-gw06 kernel: Node 0 Normal: 24145*4kB (UEM) 37973*8kB (UEM) 1490*16kB (UEM) 10430*32kB (UEM) 2260*64kB (UEM) 72*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 911820kB Aug 15 00:53:09 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 00:53:09 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 00:53:09 oak-gw06 kernel: 2089857 total pagecache pages Aug 15 00:53:09 oak-gw06 kernel: 16 pages in swap cache Aug 15 00:53:09 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 00:53:09 oak-gw06 kernel: Free swap = 4194036kB Aug 15 00:53:09 oak-gw06 kernel: Total swap = 4194300kB Aug 15 00:53:09 oak-gw06 kernel: 4194203 pages RAM Aug 15 00:53:09 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 00:53:09 oak-gw06 kernel: 127313 pages reserved Aug 15 00:53:09 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 15 00:53:09 oak-gw06 kernel: CPU: 6 PID: 5238 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 00:53:09 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 00:53:09 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 00:53:09 oak-gw06 kernel: 00000000000080d0 0000000001751c7d ffff8803f826b808 ffffffff8168662f Aug 15 00:53:09 oak-gw06 kernel: ffff8803f826b898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 00:53:09 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803f826b868 0000000001751c7d Aug 15 00:53:09 oak-gw06 kernel: Call Trace: Aug 15 00:53:09 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 00:53:09 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 00:53:09 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 00:53:09 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 00:53:09 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 00:53:09 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 00:53:09 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 00:53:09 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 00:53:09 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 00:53:09 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 00:53:09 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 00:53:09 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 00:53:09 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 00:53:09 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 00:53:09 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 00:53:09 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 00:53:09 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 00:53:09 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 00:53:09 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 00:53:09 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 00:53:09 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 00:53:09 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 00:53:09 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 00:53:09 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 00:53:09 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 00:53:09 oak-gw06 kernel: Mem-Info: Aug 15 00:53:09 oak-gw06 kernel: active_anon:24135 inactive_anon:51096 isolated_anon:0#012 active_file:539053 inactive_file:2209523 isolated_file:0#012 unevictable:0 dirty:15192 writeback:5446 unstable:0#012 slab_reclaimable:34017 slab_unreclaimable:714969#012 mapped:10577 shmem:45078 pagetables:1679 bounce:0#012 free:350573 free_pcp:1098 free_cma:0 Aug 15 00:53:09 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 00:53:09 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 00:53:09 oak-gw06 kernel: Node 0 DMA32 free:465336kB min:69724kB low:87152kB high:104584kB active_anon:13252kB inactive_anon:35588kB active_file:398332kB inactive_file:1475488kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:11944kB writeback:2840kB mapped:4792kB shmem:31268kB slab_reclaimable:18736kB slab_unreclaimable:386188kB kernel_stack:944kB pagetables:1076kB unstable:0kB bounce:0kB free_pcp:1780kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 00:53:09 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 00:53:09 oak-gw06 kernel: Node 0 Normal free:1034992kB min:323104kB low:403880kB high:484656kB active_anon:83288kB inactive_anon:168796kB active_file:1757880kB inactive_file:7236560kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:48824kB writeback:20884kB mapped:37516kB shmem:149044kB slab_reclaimable:117332kB slab_unreclaimable:2473408kB kernel_stack:4752kB pagetables:5640kB unstable:0kB bounce:0kB free_pcp:3376kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 00:53:09 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 00:53:09 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 00:53:09 oak-gw06 kernel: Node 0 DMA32: 8290*4kB (UEM) 7211*8kB (UEM) 1535*16kB (UEM) 6065*32kB (UEM) 2073*64kB (UM) 270*128kB (UM) 32*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 484912kB Aug 15 00:53:09 oak-gw06 kernel: Node 0 Normal: 35362*4kB (UEM) 44908*8kB (UEM) 4119*16kB (UEM) 10295*32kB (UEM) 2264*64kB (UEM) 72*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1050168kB Aug 15 00:53:09 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 00:53:09 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 00:53:09 oak-gw06 kernel: 2094543 total pagecache pages Aug 15 00:53:09 oak-gw06 kernel: 16 pages in swap cache Aug 15 00:53:09 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 00:53:09 oak-gw06 kernel: Free swap = 4194036kB Aug 15 00:53:09 oak-gw06 kernel: Total swap = 4194300kB Aug 15 00:53:09 oak-gw06 kernel: 4194203 pages RAM Aug 15 00:53:09 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 00:53:09 oak-gw06 kernel: 127313 pages reserved Aug 15 00:58:10 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 15 00:58:10 oak-gw06 kernel: CPU: 2 PID: 5238 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 00:58:10 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 00:58:10 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 00:58:10 oak-gw06 kernel: 00000000000080d0 0000000001751c7d ffff8803f826b858 ffffffff8168662f Aug 15 00:58:10 oak-gw06 kernel: ffff8803f826b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 00:58:10 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803f826b8b8 0000000001751c7d Aug 15 00:58:10 oak-gw06 kernel: Call Trace: Aug 15 00:58:10 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 00:58:10 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 00:58:10 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 00:58:10 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 00:58:10 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 00:58:10 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 00:58:10 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 00:58:10 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 00:58:10 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 00:58:10 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 00:58:10 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 00:58:10 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 00:58:10 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 00:58:10 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 00:58:10 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 00:58:10 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 00:58:10 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 00:58:10 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 00:58:10 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 00:58:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 00:58:10 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 00:58:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 00:58:10 oak-gw06 kernel: Mem-Info: Aug 15 00:58:10 oak-gw06 kernel: active_anon:23460 inactive_anon:51096 isolated_anon:0#012 active_file:494363 inactive_file:2515099 isolated_file:0#012 unevictable:0 dirty:3466 writeback:825 unstable:0#012 slab_reclaimable:33393 slab_unreclaimable:644403#012 mapped:10589 shmem:45078 pagetables:1688 bounce:0#012 free:210430 free_pcp:756 free_cma:0 Aug 15 00:58:10 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 00:58:10 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 00:58:10 oak-gw06 kernel: Node 0 DMA32 free:180892kB min:69724kB low:87152kB high:104584kB active_anon:11364kB inactive_anon:35588kB active_file:394692kB inactive_file:1832124kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3544kB writeback:84kB mapped:4796kB shmem:31268kB slab_reclaimable:18584kB slab_unreclaimable:361364kB kernel_stack:960kB pagetables:1076kB unstable:0kB bounce:0kB free_pcp:656kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 00:58:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 00:58:10 oak-gw06 kernel: Node 0 Normal free:636352kB min:323104kB low:403880kB high:484656kB active_anon:86180kB inactive_anon:168796kB active_file:1584864kB inactive_file:8232068kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:11328kB writeback:808kB mapped:37560kB shmem:149044kB slab_reclaimable:114968kB slab_unreclaimable:2216300kB kernel_stack:4736kB pagetables:5664kB unstable:0kB bounce:0kB free_pcp:2448kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 00:58:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 00:58:10 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 00:58:10 oak-gw06 kernel: Node 0 DMA32: 6140*4kB (UEM) 7266*8kB (UEM) 1829*16kB (UEM) 1437*32kB (UEM) 374*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 181872kB Aug 15 00:58:10 oak-gw06 kernel: Node 0 Normal: 20439*4kB (UEM) 32426*8kB (UEM) 6980*16kB (UEM) 5097*32kB (UEM) 211*64kB (UM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 629580kB Aug 15 00:58:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 00:58:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 00:58:10 oak-gw06 kernel: 2094444 total pagecache pages Aug 15 00:58:10 oak-gw06 kernel: 16 pages in swap cache Aug 15 00:58:10 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 00:58:10 oak-gw06 kernel: Free swap = 4194036kB Aug 15 00:58:10 oak-gw06 kernel: Total swap = 4194300kB Aug 15 00:58:10 oak-gw06 kernel: 4194203 pages RAM Aug 15 00:58:10 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 00:58:10 oak-gw06 kernel: 127313 pages reserved Aug 15 00:58:10 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 15 00:58:10 oak-gw06 kernel: CPU: 6 PID: 5238 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 00:58:10 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 00:58:10 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 00:58:10 oak-gw06 kernel: 00000000000080d0 0000000001751c7d ffff8803f826b808 ffffffff8168662f Aug 15 00:58:10 oak-gw06 kernel: ffff8803f826b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 00:58:10 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803f826b868 0000000001751c7d Aug 15 00:58:10 oak-gw06 kernel: Call Trace: Aug 15 00:58:10 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 00:58:10 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 00:58:10 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 00:58:10 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 00:58:10 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 00:58:10 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 00:58:10 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 00:58:10 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 00:58:10 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 00:58:10 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 00:58:10 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 00:58:10 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 00:58:10 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 00:58:10 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 00:58:10 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 00:58:10 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 00:58:10 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 00:58:10 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 00:58:10 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 00:58:10 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 00:58:10 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 00:58:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 00:58:10 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 00:58:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 00:58:10 oak-gw06 kernel: Mem-Info: Aug 15 00:58:10 oak-gw06 kernel: active_anon:24386 inactive_anon:51096 isolated_anon:0#012 active_file:495693 inactive_file:2521305 isolated_file:0#012 unevictable:0 dirty:3725 writeback:1421 unstable:0#012 slab_reclaimable:33373 slab_unreclaimable:644510#012 mapped:10589 shmem:45078 pagetables:1685 bounce:0#012 free:201786 free_pcp:832 free_cma:0 Aug 15 00:58:10 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 00:58:10 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 00:58:10 oak-gw06 kernel: Node 0 DMA32 free:183264kB min:69724kB low:87152kB high:104584kB active_anon:11364kB inactive_anon:35588kB active_file:394392kB inactive_file:1831364kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3624kB writeback:0kB mapped:4796kB shmem:31268kB slab_reclaimable:18576kB slab_unreclaimable:361652kB kernel_stack:960kB pagetables:1076kB unstable:0kB bounce:0kB free_pcp:1164kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 00:58:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 00:58:10 oak-gw06 kernel: Node 0 Normal free:599092kB min:323104kB low:403880kB high:484656kB active_anon:86180kB inactive_anon:168796kB active_file:1590720kB inactive_file:8260616kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:12052kB writeback:4520kB mapped:37560kB shmem:149044kB slab_reclaimable:114916kB slab_unreclaimable:2216372kB kernel_stack:4736kB pagetables:5664kB unstable:0kB bounce:0kB free_pcp:3348kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 00:58:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 00:58:10 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 00:58:10 oak-gw06 kernel: Node 0 DMA32: 6546*4kB (UEM) 7369*8kB (UEM) 1905*16kB (UEM) 1454*32kB (UEM) 375*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 186144kB Aug 15 00:58:10 oak-gw06 kernel: Node 0 Normal: 21472*4kB (UEM) 28418*8kB (UEM) 7000*16kB (UEM) 5207*32kB (UEM) 217*64kB (UM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 605872kB Aug 15 00:58:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 00:58:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 00:58:10 oak-gw06 kernel: 2095869 total pagecache pages Aug 15 00:58:10 oak-gw06 kernel: 16 pages in swap cache Aug 15 00:58:10 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 00:58:10 oak-gw06 kernel: Free swap = 4194036kB Aug 15 00:58:10 oak-gw06 kernel: Total swap = 4194300kB Aug 15 00:58:10 oak-gw06 kernel: 4194203 pages RAM Aug 15 00:58:10 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 00:58:10 oak-gw06 kernel: 127313 pages reserved Aug 15 01:03:10 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 01:03:10 oak-gw06 kernel: CPU: 0 PID: 5249 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:03:10 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:03:10 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:03:10 oak-gw06 kernel: 00000000000080d0 00000000fcf33796 ffff880040bff858 ffffffff8168662f Aug 15 01:03:10 oak-gw06 kernel: ffff880040bff8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:03:10 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880040bff8b8 00000000fcf33796 Aug 15 01:03:10 oak-gw06 kernel: Call Trace: Aug 15 01:03:10 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:03:10 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:03:10 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:03:10 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:03:10 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 01:03:10 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 01:03:10 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:03:10 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:03:10 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:03:10 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:03:10 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:03:10 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:03:10 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:03:10 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:03:10 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:03:10 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:03:10 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:03:10 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:03:10 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:03:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:03:10 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:03:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:03:10 oak-gw06 kernel: Mem-Info: Aug 15 01:03:10 oak-gw06 kernel: active_anon:24387 inactive_anon:51094 isolated_anon:0#012 active_file:409409 inactive_file:2349169 isolated_file:0#012 unevictable:0 dirty:3794 writeback:1606 unstable:0#012 slab_reclaimable:32990 slab_unreclaimable:614872#012 mapped:10607 shmem:45078 pagetables:1685 bounce:0#012 free:488954 free_pcp:611 free_cma:0 Aug 15 01:03:10 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:03:10 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:03:10 oak-gw06 kernel: Node 0 DMA32 free:464504kB min:69724kB low:87152kB high:104584kB active_anon:11372kB inactive_anon:35584kB active_file:321092kB inactive_file:1644864kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4100kB writeback:1504kB mapped:4808kB shmem:31268kB slab_reclaimable:18488kB slab_unreclaimable:342812kB kernel_stack:960kB pagetables:1076kB unstable:0kB bounce:0kB free_pcp:468kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:03:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:03:10 oak-gw06 kernel: Node 0 Normal free:1458352kB min:323104kB low:403880kB high:484656kB active_anon:86176kB inactive_anon:168792kB active_file:1316544kB inactive_file:7771752kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:11804kB writeback:2544kB mapped:37620kB shmem:149044kB slab_reclaimable:113472kB slab_unreclaimable:2116660kB kernel_stack:4768kB pagetables:5664kB unstable:0kB bounce:0kB free_pcp:2408kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:03:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:03:10 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:03:10 oak-gw06 kernel: Node 0 DMA32: 6903*4kB (UEM) 6980*8kB (UE) 1913*16kB (UE) 4419*32kB (UEM) 2730*64kB (UEM) 175*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 453100kB Aug 15 01:03:10 oak-gw06 kernel: Node 0 Normal: 38537*4kB (UEM) 37222*8kB (UEM) 7657*16kB (UEM) 20130*32kB (UEM) 3509*64kB (UEM) 133*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1460452kB Aug 15 01:03:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:03:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:03:10 oak-gw06 kernel: 2097786 total pagecache pages Aug 15 01:03:10 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:03:10 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:03:10 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:03:10 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:03:10 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:03:10 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:03:10 oak-gw06 kernel: 127313 pages reserved Aug 15 01:03:10 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 01:03:10 oak-gw06 kernel: CPU: 0 PID: 5249 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:03:10 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:03:10 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:03:10 oak-gw06 kernel: 00000000000080d0 00000000fcf33796 ffff880040bff808 ffffffff8168662f Aug 15 01:03:10 oak-gw06 kernel: ffff880040bff898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 01:03:10 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880040bff868 00000000fcf33796 Aug 15 01:03:10 oak-gw06 kernel: Call Trace: Aug 15 01:03:10 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:03:10 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:03:10 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 01:03:10 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:03:10 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:03:10 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 01:03:10 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 01:03:10 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 01:03:10 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 01:03:10 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:03:10 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:03:10 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:03:10 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:03:10 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:03:10 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:03:10 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:03:10 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:03:10 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:03:10 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:03:10 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:03:10 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:03:10 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:03:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:03:10 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:03:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:03:10 oak-gw06 kernel: Mem-Info: Aug 15 01:03:10 oak-gw06 kernel: active_anon:24387 inactive_anon:51094 isolated_anon:0#012 active_file:409409 inactive_file:2349409 isolated_file:0#012 unevictable:0 dirty:3697 writeback:1588 unstable:0#012 slab_reclaimable:32990 slab_unreclaimable:614872#012 mapped:10607 shmem:45078 pagetables:1685 bounce:0#012 free:487355 free_pcp:1201 free_cma:0 Aug 15 01:03:10 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:03:10 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:03:10 oak-gw06 kernel: Node 0 DMA32 free:462248kB min:69724kB low:87152kB high:104584kB active_anon:11372kB inactive_anon:35584kB active_file:321092kB inactive_file:1641840kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2596kB writeback:0kB mapped:4808kB shmem:31268kB slab_reclaimable:18488kB slab_unreclaimable:342812kB kernel_stack:960kB pagetables:1076kB unstable:0kB bounce:0kB free_pcp:1784kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:03:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:03:10 oak-gw06 kernel: Node 0 Normal free:1465584kB min:323104kB low:403880kB high:484656kB active_anon:86176kB inactive_anon:168792kB active_file:1316544kB inactive_file:7753032kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:12192kB writeback:2932kB mapped:37620kB shmem:149044kB slab_reclaimable:113472kB slab_unreclaimable:2116660kB kernel_stack:4768kB pagetables:5664kB unstable:0kB bounce:0kB free_pcp:3792kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:03:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:03:10 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:03:10 oak-gw06 kernel: Node 0 DMA32: 7456*4kB (UEM) 7042*8kB (UEM) 2163*16kB (UEM) 4590*32kB (UEM) 2730*64kB (UEM) 175*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 465280kB Aug 15 01:03:10 oak-gw06 kernel: Node 0 Normal: 35768*4kB (UE) 37224*8kB (UEM) 8017*16kB (UEM) 20128*32kB (UEM) 3509*64kB (UEM) 133*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1455088kB Aug 15 01:03:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:03:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:03:10 oak-gw06 kernel: 2097113 total pagecache pages Aug 15 01:03:10 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:03:10 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:03:10 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:03:10 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:03:10 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:03:10 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:03:10 oak-gw06 kernel: 127313 pages reserved Aug 15 01:08:10 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 01:08:10 oak-gw06 kernel: CPU: 2 PID: 5249 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:08:10 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:08:10 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:08:10 oak-gw06 kernel: 00000000000080d0 00000000fcf33796 ffff880040bff858 ffffffff8168662f Aug 15 01:08:10 oak-gw06 kernel: ffff880040bff8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:08:10 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880040bff8b8 00000000fcf33796 Aug 15 01:08:10 oak-gw06 kernel: Call Trace: Aug 15 01:08:10 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:08:10 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:08:10 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:08:10 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:08:10 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 01:08:10 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 01:08:10 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:08:10 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:08:10 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:08:10 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:08:10 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:08:10 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:08:10 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:08:10 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:08:10 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:08:10 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:08:10 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:08:10 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:08:10 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:08:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:08:10 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:08:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:08:10 oak-gw06 kernel: Mem-Info: Aug 15 01:08:10 oak-gw06 kernel: active_anon:25136 inactive_anon:51094 isolated_anon:0#012 active_file:508318 inactive_file:2392970 isolated_file:0#012 unevictable:0 dirty:3741 writeback:1951 unstable:0#012 slab_reclaimable:32823 slab_unreclaimable:601495#012 mapped:10615 shmem:45078 pagetables:1683 bounce:0#012 free:312583 free_pcp:1209 free_cma:0 Aug 15 01:08:10 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:08:10 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:08:10 oak-gw06 kernel: Node 0 DMA32 free:421372kB min:69724kB low:87152kB high:104584kB active_anon:19876kB inactive_anon:35584kB active_file:381980kB inactive_file:1584380kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2588kB writeback:0kB mapped:4804kB shmem:31268kB slab_reclaimable:18408kB slab_unreclaimable:337392kB kernel_stack:1024kB pagetables:1936kB unstable:0kB bounce:0kB free_pcp:3696kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:08:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:08:10 oak-gw06 kernel: Node 0 Normal free:839012kB min:323104kB low:403880kB high:484656kB active_anon:80668kB inactive_anon:168792kB active_file:1651292kB inactive_file:7961788kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:13492kB writeback:10908kB mapped:37656kB shmem:149044kB slab_reclaimable:112884kB slab_unreclaimable:2067788kB kernel_stack:4656kB pagetables:4796kB unstable:0kB bounce:0kB free_pcp:2956kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:08:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:08:10 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:08:10 oak-gw06 kernel: Node 0 DMA32: 2490*4kB (UEM) 6618*8kB (UEM) 1349*16kB (UEM) 6049*32kB (UEM) 2009*64kB (UM) 77*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 416488kB Aug 15 01:08:10 oak-gw06 kernel: Node 0 Normal: 12150*4kB (UEM) 26589*8kB (UEM) 3310*16kB (UEM) 12126*32kB (UEM) 2274*64kB (UEM) 65*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 856160kB Aug 15 01:08:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:08:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:08:10 oak-gw06 kernel: 2088296 total pagecache pages Aug 15 01:08:10 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:08:10 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:08:10 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:08:10 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:08:10 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:08:10 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:08:10 oak-gw06 kernel: 127313 pages reserved Aug 15 01:08:10 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 01:08:10 oak-gw06 kernel: CPU: 7 PID: 5249 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:08:10 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:08:10 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:08:10 oak-gw06 kernel: 00000000000080d0 00000000fcf33796 ffff880040bff808 ffffffff8168662f Aug 15 01:08:10 oak-gw06 kernel: ffff880040bff898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:08:10 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880040bff868 00000000fcf33796 Aug 15 01:08:10 oak-gw06 kernel: Call Trace: Aug 15 01:08:10 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:08:10 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:08:10 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:08:10 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:08:10 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 01:08:10 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 01:08:10 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 01:08:10 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 01:08:10 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:08:10 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:08:10 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:08:10 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:08:10 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:08:10 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:08:10 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:08:10 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:08:10 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:08:10 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:08:10 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:08:10 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:08:10 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:08:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:08:10 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:08:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:08:10 oak-gw06 kernel: Mem-Info: Aug 15 01:08:10 oak-gw06 kernel: active_anon:25136 inactive_anon:51094 isolated_anon:0#012 active_file:508448 inactive_file:2386761 isolated_file:0#012 unevictable:0 dirty:4226 writeback:3879 unstable:0#012 slab_reclaimable:32823 slab_unreclaimable:601299#012 mapped:10615 shmem:45078 pagetables:1683 bounce:0#012 free:318025 free_pcp:945 free_cma:0 Aug 15 01:08:10 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:08:10 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:08:10 oak-gw06 kernel: Node 0 DMA32 free:422068kB min:69724kB low:87152kB high:104584kB active_anon:19876kB inactive_anon:35584kB active_file:381980kB inactive_file:1586900kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2588kB writeback:1504kB mapped:4804kB shmem:31268kB slab_reclaimable:18408kB slab_unreclaimable:337392kB kernel_stack:1024kB pagetables:1936kB unstable:0kB bounce:0kB free_pcp:1012kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:08:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:08:10 oak-gw06 kernel: Node 0 Normal free:831120kB min:323104kB low:403880kB high:484656kB active_anon:80668kB inactive_anon:168792kB active_file:1651812kB inactive_file:7969068kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:17372kB writeback:15952kB mapped:37656kB shmem:149044kB slab_reclaimable:112884kB slab_unreclaimable:2067788kB kernel_stack:4656kB pagetables:4796kB unstable:0kB bounce:0kB free_pcp:3072kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:08:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:08:10 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:08:10 oak-gw06 kernel: Node 0 DMA32: 2788*4kB (UEM) 5941*8kB (UEM) 1311*16kB (UEM) 6139*32kB (UEM) 2010*64kB (UM) 77*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 414600kB Aug 15 01:08:10 oak-gw06 kernel: Node 0 Normal: 10600*4kB (UE) 26358*8kB (UE) 1130*16kB (UEM) 11936*32kB (UEM) 2274*64kB (UEM) 65*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 807152kB Aug 15 01:08:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:08:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:08:10 oak-gw06 kernel: 2100688 total pagecache pages Aug 15 01:08:10 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:08:10 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:08:10 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:08:10 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:08:10 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:08:10 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:08:10 oak-gw06 kernel: 127313 pages reserved Aug 15 01:13:10 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 01:13:10 oak-gw06 kernel: CPU: 6 PID: 5249 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:13:10 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:13:10 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:13:10 oak-gw06 kernel: 00000000000080d0 00000000fcf33796 ffff880040bff858 ffffffff8168662f Aug 15 01:13:10 oak-gw06 kernel: ffff880040bff8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:13:10 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880040bff8b8 00000000fcf33796 Aug 15 01:13:10 oak-gw06 kernel: Call Trace: Aug 15 01:13:10 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:13:10 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:13:10 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:13:10 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:13:10 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 01:13:10 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 01:13:10 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:13:10 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:13:10 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:13:10 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:13:10 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:13:10 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:13:10 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:13:10 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:13:10 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:13:10 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:13:10 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:13:10 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:13:10 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:13:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:13:10 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:13:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:13:10 oak-gw06 kernel: Mem-Info: Aug 15 01:13:10 oak-gw06 kernel: active_anon:20238 inactive_anon:51094 isolated_anon:0#012 active_file:523102 inactive_file:1956557 isolated_file:0#012 unevictable:0 dirty:8560 writeback:4255 unstable:0#012 slab_reclaimable:32785 slab_unreclaimable:601878#012 mapped:10622 shmem:45078 pagetables:1662 bounce:0#012 free:773287 free_pcp:662 free_cma:0 Aug 15 01:13:10 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:13:10 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:13:10 oak-gw06 kernel: Node 0 DMA32 free:661600kB min:69724kB low:87152kB high:104584kB active_anon:10704kB inactive_anon:35584kB active_file:399480kB inactive_file:1357728kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:8840kB writeback:292kB mapped:4812kB shmem:31268kB slab_reclaimable:18392kB slab_unreclaimable:338536kB kernel_stack:976kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:13:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:13:10 oak-gw06 kernel: Node 0 Normal free:2408680kB min:323104kB low:403880kB high:484656kB active_anon:70508kB inactive_anon:168792kB active_file:1692928kB inactive_file:6472660kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:24624kB writeback:18668kB mapped:37676kB shmem:149044kB slab_reclaimable:112748kB slab_unreclaimable:2068960kB kernel_stack:4736kB pagetables:5588kB unstable:0kB bounce:0kB free_pcp:2652kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:13:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:13:10 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:13:10 oak-gw06 kernel: Node 0 DMA32: 6736*4kB (UEM) 22489*8kB (UEM) 12129*16kB (UEM) 6678*32kB (UEM) 762*64kB (UM) 10*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 664664kB Aug 15 01:13:10 oak-gw06 kernel: Node 0 Normal: 18990*4kB (UE) 86409*8kB (UEM) 61839*16kB (UEM) 16632*32kB (UEM) 1697*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2402352kB Aug 15 01:13:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:13:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:13:10 oak-gw06 kernel: 2101707 total pagecache pages Aug 15 01:13:10 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:13:10 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:13:10 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:13:10 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:13:10 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:13:10 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:13:10 oak-gw06 kernel: 127313 pages reserved Aug 15 01:13:10 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 01:13:10 oak-gw06 kernel: CPU: 6 PID: 5249 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:13:10 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:13:10 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:13:10 oak-gw06 kernel: 00000000000080d0 00000000fcf33796 ffff880040bff808 ffffffff8168662f Aug 15 01:13:10 oak-gw06 kernel: ffff880040bff898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 01:13:10 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880040bff868 00000000fcf33796 Aug 15 01:13:10 oak-gw06 kernel: Call Trace: Aug 15 01:13:10 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:13:10 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:13:10 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 01:13:10 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:13:10 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:13:10 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 01:13:10 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 01:13:10 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 01:13:10 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 01:13:10 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:13:10 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:13:10 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:13:10 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:13:10 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:13:10 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:13:10 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:13:10 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:13:10 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:13:10 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:13:10 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:13:10 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:13:10 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:13:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:13:10 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:13:10 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:13:10 oak-gw06 kernel: Mem-Info: Aug 15 01:13:10 oak-gw06 kernel: active_anon:20238 inactive_anon:51094 isolated_anon:0#012 active_file:513749 inactive_file:1951812 isolated_file:0#012 unevictable:0 dirty:9724 writeback:5322 unstable:0#012 slab_reclaimable:32785 slab_unreclaimable:601878#012 mapped:10622 shmem:45078 pagetables:1662 bounce:0#012 free:784900 free_pcp:1178 free_cma:0 Aug 15 01:13:10 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:13:10 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:13:10 oak-gw06 kernel: Node 0 DMA32 free:676768kB min:69724kB low:87152kB high:104584kB active_anon:10704kB inactive_anon:35584kB active_file:387384kB inactive_file:1357224kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4328kB writeback:4804kB mapped:4812kB shmem:31268kB slab_reclaimable:18392kB slab_unreclaimable:338536kB kernel_stack:976kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:2088kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:13:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:13:10 oak-gw06 kernel: Node 0 Normal free:2462344kB min:323104kB low:403880kB high:484656kB active_anon:70248kB inactive_anon:168792kB active_file:1650808kB inactive_file:6452640kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:33936kB writeback:15952kB mapped:37676kB shmem:149044kB slab_reclaimable:112748kB slab_unreclaimable:2068960kB kernel_stack:4736kB pagetables:5588kB unstable:0kB bounce:0kB free_pcp:3940kB local_pcp:4kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:13:10 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:13:10 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:13:10 oak-gw06 kernel: Node 0 DMA32: 7192*4kB (UEM) 23482*8kB (UEM) 12645*16kB (UEM) 6850*32kB (UEM) 778*64kB (UM) 10*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 689216kB Aug 15 01:13:10 oak-gw06 kernel: Node 0 Normal: 19030*4kB (UE) 88844*8kB (UEM) 63154*16kB (UEM) 17247*32kB (UEM) 1711*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2463608kB Aug 15 01:13:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:13:10 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:13:10 oak-gw06 kernel: 2079114 total pagecache pages Aug 15 01:13:10 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:13:10 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:13:10 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:13:10 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:13:10 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:13:10 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:13:10 oak-gw06 kernel: 127313 pages reserved Aug 15 01:18:10 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 15 01:18:10 oak-gw06 kernel: CPU: 6 PID: 5305 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:18:10 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:18:10 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:18:10 oak-gw06 kernel: 00000000000080d0 00000000921eaacc ffff8800735df858 ffffffff8168662f Aug 15 01:18:10 oak-gw06 kernel: ffff8800735df8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:18:10 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800735df8b8 00000000921eaacc Aug 15 01:18:10 oak-gw06 kernel: Call Trace: Aug 15 01:18:10 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:18:10 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:18:10 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:18:10 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:18:10 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 01:18:10 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 01:18:10 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:18:10 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:18:10 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:18:10 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:18:10 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:18:10 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:18:10 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:18:10 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:18:10 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:18:10 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:18:11 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:18:11 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:18:11 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:18:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:18:11 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:18:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:18:11 oak-gw06 kernel: Mem-Info: Aug 15 01:18:11 oak-gw06 kernel: active_anon:24418 inactive_anon:51094 isolated_anon:0#012 active_file:266512 inactive_file:2241354 isolated_file:0#012 unevictable:0 dirty:3883 writeback:1310 unstable:0#012 slab_reclaimable:32777 slab_unreclaimable:597943#012 mapped:10640 shmem:45078 pagetables:1689 bounce:0#012 free:757674 free_pcp:1519 free_cma:0 Aug 15 01:18:11 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:18:11 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:18:11 oak-gw06 kernel: Node 0 DMA32 free:742692kB min:69724kB low:87152kB high:104584kB active_anon:11284kB inactive_anon:35584kB active_file:187468kB inactive_file:1502048kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2592kB writeback:116kB mapped:4820kB shmem:31268kB slab_reclaimable:18392kB slab_unreclaimable:333368kB kernel_stack:960kB pagetables:1068kB unstable:0kB bounce:0kB free_pcp:3416kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:18:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:18:11 oak-gw06 kernel: Node 0 Normal free:2259324kB min:323104kB low:403880kB high:484656kB active_anon:86388kB inactive_anon:168792kB active_file:878580kB inactive_file:7475328kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:12940kB writeback:4736kB mapped:37740kB shmem:149044kB slab_reclaimable:112716kB slab_unreclaimable:2058388kB kernel_stack:4736kB pagetables:5688kB unstable:0kB bounce:0kB free_pcp:3052kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:18:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:18:11 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:18:11 oak-gw06 kernel: Node 0 DMA32: 7273*4kB (UEM) 8486*8kB (UEM) 12355*16kB (UEM) 6057*32kB (UEM) 3348*64kB (UEM) 331*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 745892kB Aug 15 01:18:11 oak-gw06 kernel: Node 0 Normal: 33464*4kB (UEM) 43961*8kB (UEM) 51641*16kB (UEM) 22310*32kB (UEM) 3285*64kB (UEM) 130*128kB (UM) 2*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2253112kB Aug 15 01:18:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:18:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:18:11 oak-gw06 kernel: 2071718 total pagecache pages Aug 15 01:18:11 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:18:11 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:18:11 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:18:11 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:18:11 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:18:11 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:18:11 oak-gw06 kernel: 127313 pages reserved Aug 15 01:18:11 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 15 01:18:11 oak-gw06 kernel: CPU: 6 PID: 5305 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:18:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:18:11 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:18:11 oak-gw06 kernel: 00000000000080d0 00000000921eaacc ffff8800735df808 ffffffff8168662f Aug 15 01:18:11 oak-gw06 kernel: ffff8800735df898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:18:11 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800735df868 00000000921eaacc Aug 15 01:18:11 oak-gw06 kernel: Call Trace: Aug 15 01:18:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:18:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:18:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:18:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:18:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 01:18:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 01:18:11 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 01:18:11 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 01:18:11 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:18:11 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:18:11 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:18:11 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:18:11 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:18:11 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:18:11 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:18:11 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:18:11 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:18:11 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:18:11 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:18:11 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:18:11 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:18:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:18:11 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:18:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:18:11 oak-gw06 kernel: Mem-Info: Aug 15 01:18:11 oak-gw06 kernel: active_anon:24418 inactive_anon:51094 isolated_anon:0#012 active_file:266512 inactive_file:2250324 isolated_file:0#012 unevictable:0 dirty:3592 writeback:1892 unstable:0#012 slab_reclaimable:32777 slab_unreclaimable:597943#012 mapped:10640 shmem:45078 pagetables:1689 bounce:0#012 free:749495 free_pcp:672 free_cma:0 Aug 15 01:18:11 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:18:11 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:18:11 oak-gw06 kernel: Node 0 DMA32 free:750968kB min:69724kB low:87152kB high:104584kB active_anon:11284kB inactive_anon:35584kB active_file:187388kB inactive_file:1501840kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2636kB writeback:80kB mapped:4820kB shmem:31268kB slab_reclaimable:18392kB slab_unreclaimable:333368kB kernel_stack:960kB pagetables:1068kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:18:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:18:11 oak-gw06 kernel: Node 0 Normal free:2215936kB min:323104kB low:403880kB high:484656kB active_anon:86648kB inactive_anon:168792kB active_file:878448kB inactive_file:7513960kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:13332kB writeback:7316kB mapped:37740kB shmem:149044kB slab_reclaimable:112716kB slab_unreclaimable:2058172kB kernel_stack:4736kB pagetables:5688kB unstable:0kB bounce:0kB free_pcp:2884kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:18:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:18:11 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:18:11 oak-gw06 kernel: Node 0 DMA32: 7789*4kB (UEM) 8487*8kB (UEM) 12667*16kB (UEM) 6061*32kB (UEM) 3348*64kB (UEM) 331*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 753084kB Aug 15 01:18:11 oak-gw06 kernel: Node 0 Normal: 33303*4kB (UEM) 39126*8kB (UEM) 51433*16kB (UEM) 22312*32kB (UEM) 3285*64kB (UEM) 130*128kB (UM) 2*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2210524kB Aug 15 01:18:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:18:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:18:11 oak-gw06 kernel: 2082807 total pagecache pages Aug 15 01:18:11 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:18:11 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:18:11 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:18:11 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:18:11 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:18:11 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:18:11 oak-gw06 kernel: 127313 pages reserved Aug 15 01:23:11 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 01:23:11 oak-gw06 kernel: CPU: 6 PID: 5330 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:23:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:23:11 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:23:11 oak-gw06 kernel: 00000000000080d0 000000007e5cf650 ffff8800afe77858 ffffffff8168662f Aug 15 01:23:11 oak-gw06 kernel: ffff8800afe778e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:23:11 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800afe778b8 000000007e5cf650 Aug 15 01:23:11 oak-gw06 kernel: Call Trace: Aug 15 01:23:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:23:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:23:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:23:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:23:11 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 01:23:11 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 01:23:11 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:23:11 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:23:11 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:23:11 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:23:11 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:23:11 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:23:11 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:23:11 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:23:11 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:23:11 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:23:11 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:23:11 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:23:11 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:23:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:23:11 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:23:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:23:11 oak-gw06 kernel: Mem-Info: Aug 15 01:23:11 oak-gw06 kernel: active_anon:19784 inactive_anon:51094 isolated_anon:0#012 active_file:332473 inactive_file:2399841 isolated_file:0#012 unevictable:0 dirty:4146 writeback:5801 unstable:0#012 slab_reclaimable:32768 slab_unreclaimable:598346#012 mapped:10645 shmem:45078 pagetables:1686 bounce:0#012 free:536611 free_pcp:974 free_cma:0 Aug 15 01:23:11 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:23:11 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:23:11 oak-gw06 kernel: Node 0 DMA32 free:607468kB min:69724kB low:87152kB high:104584kB active_anon:10472kB inactive_anon:35584kB active_file:236236kB inactive_file:1596884kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2792kB writeback:784kB mapped:4824kB shmem:31268kB slab_reclaimable:18384kB slab_unreclaimable:331712kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:1256kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:23:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:23:11 oak-gw06 kernel: Node 0 Normal free:1515280kB min:323104kB low:403880kB high:484656kB active_anon:68664kB inactive_anon:168792kB active_file:1093656kB inactive_file:8013920kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:16896kB writeback:26300kB mapped:37756kB shmem:149044kB slab_reclaimable:112688kB slab_unreclaimable:2061656kB kernel_stack:4768kB pagetables:5684kB unstable:0kB bounce:0kB free_pcp:2708kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:23:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:23:11 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:23:11 oak-gw06 kernel: Node 0 DMA32: 7105*4kB (UEM) 9399*8kB (UEM) 7605*16kB (UEM) 5812*32kB (UEM) 2318*64kB (UEM) 375*128kB (UM) 6*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 609164kB Aug 15 01:23:11 oak-gw06 kernel: Node 0 Normal: 23716*4kB (UEM) 30440*8kB (UEM) 31615*16kB (UEM) 17138*32kB (UEM) 1684*64kB (UEM) 9*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1501568kB Aug 15 01:23:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:23:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:23:11 oak-gw06 kernel: 2095302 total pagecache pages Aug 15 01:23:11 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:23:11 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:23:11 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:23:11 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:23:11 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:23:11 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:23:11 oak-gw06 kernel: 127313 pages reserved Aug 15 01:23:11 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 01:23:11 oak-gw06 kernel: CPU: 6 PID: 5330 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:23:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:23:11 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:23:11 oak-gw06 kernel: 00000000000080d0 000000007e5cf650 ffff8800afe77808 ffffffff8168662f Aug 15 01:23:11 oak-gw06 kernel: ffff8800afe77898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 01:23:11 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800afe77868 000000007e5cf650 Aug 15 01:23:11 oak-gw06 kernel: Call Trace: Aug 15 01:23:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:23:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:23:11 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 01:23:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:23:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:23:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 01:23:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 01:23:11 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 01:23:11 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 01:23:11 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:23:11 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:23:11 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:23:11 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:23:11 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:23:11 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:23:11 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:23:11 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:23:11 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:23:11 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:23:11 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:23:11 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:23:11 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:23:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:23:11 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:23:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:23:11 oak-gw06 kernel: Mem-Info: Aug 15 01:23:11 oak-gw06 kernel: active_anon:19784 inactive_anon:51094 isolated_anon:0#012 active_file:332473 inactive_file:2410391 isolated_file:0#012 unevictable:0 dirty:5352 writeback:4305 unstable:0#012 slab_reclaimable:32768 slab_unreclaimable:598346#012 mapped:10645 shmem:45078 pagetables:1686 bounce:0#012 free:525766 free_pcp:489 free_cma:0 Aug 15 01:23:11 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:23:11 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:23:11 oak-gw06 kernel: Node 0 DMA32 free:607400kB min:69724kB low:87152kB high:104584kB active_anon:10472kB inactive_anon:35584kB active_file:236236kB inactive_file:1599616kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2508kB writeback:1536kB mapped:4824kB shmem:31268kB slab_reclaimable:18384kB slab_unreclaimable:331712kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:992kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:23:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:23:11 oak-gw06 kernel: Node 0 Normal free:1465832kB min:323104kB low:403880kB high:484656kB active_anon:68664kB inactive_anon:168792kB active_file:1093656kB inactive_file:8052348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:22780kB writeback:17236kB mapped:37756kB shmem:149044kB slab_reclaimable:112688kB slab_unreclaimable:2061656kB kernel_stack:4768kB pagetables:5684kB unstable:0kB bounce:0kB free_pcp:1620kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:23:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:23:11 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:23:11 oak-gw06 kernel: Node 0 DMA32: 7144*4kB (UEM) 8987*8kB (UEM) 7767*16kB (UEM) 5812*32kB (UEM) 2318*64kB (UEM) 375*128kB (UM) 6*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 608616kB Aug 15 01:23:11 oak-gw06 kernel: Node 0 Normal: 22565*4kB (UEM) 26220*8kB (UEM) 30985*16kB (UEM) 17139*32kB (UEM) 1685*64kB (UEM) 9*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1453220kB Aug 15 01:23:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:23:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:23:11 oak-gw06 kernel: 2100821 total pagecache pages Aug 15 01:23:11 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:23:11 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:23:11 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:23:11 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:23:11 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:23:11 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:23:11 oak-gw06 kernel: 127313 pages reserved Aug 15 01:28:11 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 01:28:11 oak-gw06 kernel: CPU: 6 PID: 5330 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:28:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:28:11 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:28:11 oak-gw06 kernel: 00000000000080d0 000000007e5cf650 ffff8800afe77858 ffffffff8168662f Aug 15 01:28:11 oak-gw06 kernel: ffff8800afe778e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:28:11 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800afe778b8 000000007e5cf650 Aug 15 01:28:11 oak-gw06 kernel: Call Trace: Aug 15 01:28:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:28:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:28:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:28:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:28:11 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 01:28:11 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 01:28:11 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:28:11 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:28:11 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:28:11 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:28:11 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:28:11 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:28:11 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:28:11 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:28:11 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:28:11 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:28:11 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:28:11 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:28:11 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:28:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:28:11 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:28:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:28:11 oak-gw06 kernel: Mem-Info: Aug 15 01:28:11 oak-gw06 kernel: active_anon:24620 inactive_anon:51094 isolated_anon:0#012 active_file:378096 inactive_file:2296370 isolated_file:0#012 unevictable:0 dirty:4110 writeback:3337 unstable:0#012 slab_reclaimable:32759 slab_unreclaimable:595616#012 mapped:10676 shmem:45078 pagetables:1707 bounce:0#012 free:591974 free_pcp:567 free_cma:0 Aug 15 01:28:11 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:28:11 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:28:11 oak-gw06 kernel: Node 0 DMA32 free:628940kB min:69724kB low:87152kB high:104584kB active_anon:10516kB inactive_anon:35584kB active_file:272088kB inactive_file:1538000kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3424kB writeback:1460kB mapped:4828kB shmem:31268kB slab_reclaimable:18376kB slab_unreclaimable:330840kB kernel_stack:944kB pagetables:1080kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:28:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:28:11 oak-gw06 kernel: Node 0 Normal free:1716136kB min:323104kB low:403880kB high:484656kB active_anon:87964kB inactive_anon:168792kB active_file:1240296kB inactive_file:7654500kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:12240kB writeback:14992kB mapped:37876kB shmem:149044kB slab_reclaimable:112660kB slab_unreclaimable:2051608kB kernel_stack:4752kB pagetables:5748kB unstable:0kB bounce:0kB free_pcp:2928kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:28:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:28:11 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:28:11 oak-gw06 kernel: Node 0 DMA32: 7171*4kB (UEM) 6441*8kB (UEM) 3060*16kB (UEM) 6467*32kB (UEM) 3793*64kB (UEM) 405*128kB (UM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 632756kB Aug 15 01:28:11 oak-gw06 kernel: Node 0 Normal: 33115*4kB (UEM) 33157*8kB (UE) 16321*16kB (UEM) 25443*32kB (UEM) 3556*64kB (UM) 102*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1713924kB Aug 15 01:28:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:28:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:28:11 oak-gw06 kernel: 2103037 total pagecache pages Aug 15 01:28:11 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:28:11 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:28:11 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:28:11 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:28:11 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:28:11 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:28:11 oak-gw06 kernel: 127313 pages reserved Aug 15 01:28:11 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 01:28:11 oak-gw06 kernel: CPU: 6 PID: 5330 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:28:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:28:11 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:28:11 oak-gw06 kernel: 00000000000080d0 000000007e5cf650 ffff8800afe77808 ffffffff8168662f Aug 15 01:28:11 oak-gw06 kernel: ffff8800afe77898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:28:11 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800afe77868 000000007e5cf650 Aug 15 01:28:11 oak-gw06 kernel: Call Trace: Aug 15 01:28:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:28:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:28:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:28:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:28:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 01:28:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 01:28:11 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 01:28:11 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 01:28:11 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:28:11 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:28:11 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:28:11 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:28:11 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:28:11 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:28:11 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:28:11 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:28:11 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:28:11 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:28:11 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:28:11 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:28:11 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:28:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:28:11 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:28:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:28:11 oak-gw06 kernel: Mem-Info: Aug 15 01:28:11 oak-gw06 kernel: active_anon:24620 inactive_anon:51094 isolated_anon:0#012 active_file:378096 inactive_file:2301444 isolated_file:0#012 unevictable:0 dirty:3340 writeback:6241 unstable:0#012 slab_reclaimable:32759 slab_unreclaimable:595616#012 mapped:10676 shmem:45078 pagetables:1707 bounce:0#012 free:588375 free_pcp:475 free_cma:0 Aug 15 01:28:11 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:28:11 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:28:11 oak-gw06 kernel: Node 0 DMA32 free:631492kB min:69724kB low:87152kB high:104584kB active_anon:10516kB inactive_anon:35584kB active_file:272088kB inactive_file:1537496kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2672kB writeback:2212kB mapped:4828kB shmem:31268kB slab_reclaimable:18376kB slab_unreclaimable:330840kB kernel_stack:944kB pagetables:1080kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:28:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:28:11 oak-gw06 kernel: Node 0 Normal free:1700396kB min:323104kB low:403880kB high:484656kB active_anon:87964kB inactive_anon:168792kB active_file:1240296kB inactive_file:7670620kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:9912kB writeback:24692kB mapped:37876kB shmem:149044kB slab_reclaimable:112660kB slab_unreclaimable:2051608kB kernel_stack:4752kB pagetables:5748kB unstable:0kB bounce:0kB free_pcp:2160kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:28:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:28:11 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:28:11 oak-gw06 kernel: Node 0 DMA32: 7173*4kB (UEM) 6504*8kB (UEM) 3078*16kB (UEM) 6468*32kB (UEM) 3793*64kB (UEM) 405*128kB (UM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 633588kB Aug 15 01:28:11 oak-gw06 kernel: Node 0 Normal: 32828*4kB (UE) 33158*8kB (UEM) 15221*16kB (UEM) 25443*32kB (UEM) 3556*64kB (UM) 102*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1695184kB Aug 15 01:28:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:28:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:28:11 oak-gw06 kernel: 2101515 total pagecache pages Aug 15 01:28:11 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:28:11 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:28:11 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:28:11 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:28:11 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:28:11 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:28:11 oak-gw06 kernel: 127313 pages reserved Aug 15 01:33:11 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 01:33:11 oak-gw06 kernel: CPU: 6 PID: 5330 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:33:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:33:11 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:33:11 oak-gw06 kernel: 00000000000080d0 000000007e5cf650 ffff8800afe77858 ffffffff8168662f Aug 15 01:33:11 oak-gw06 kernel: ffff8800afe778e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:33:11 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800afe778b8 000000007e5cf650 Aug 15 01:33:11 oak-gw06 kernel: Call Trace: Aug 15 01:33:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:33:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:33:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:33:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:33:11 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 01:33:11 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 01:33:11 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:33:11 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:33:11 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:33:11 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:33:11 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:33:11 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:33:11 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:33:11 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:33:11 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:33:11 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:33:11 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:33:11 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:33:11 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:33:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:33:11 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:33:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:33:11 oak-gw06 kernel: Mem-Info: Aug 15 01:33:11 oak-gw06 kernel: active_anon:19323 inactive_anon:51094 isolated_anon:0#012 active_file:644118 inactive_file:1949182 isolated_file:0#012 unevictable:0 dirty:4386 writeback:2079 unstable:0#012 slab_reclaimable:32671 slab_unreclaimable:590821#012 mapped:10668 shmem:45078 pagetables:1669 bounce:0#012 free:685189 free_pcp:446 free_cma:0 Aug 15 01:33:11 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:33:11 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:33:11 oak-gw06 kernel: Node 0 DMA32 free:613984kB min:69724kB low:87152kB high:104584kB active_anon:9840kB inactive_anon:35584kB active_file:481240kB inactive_file:1348188kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2932kB writeback:448kB mapped:4824kB shmem:31268kB slab_reclaimable:18312kB slab_unreclaimable:331128kB kernel_stack:960kB pagetables:1072kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:33:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:33:11 oak-gw06 kernel: Node 0 Normal free:2106188kB min:323104kB low:403880kB high:484656kB active_anon:67712kB inactive_anon:168792kB active_file:2104820kB inactive_file:6443892kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:14612kB writeback:7480kB mapped:37848kB shmem:149044kB slab_reclaimable:112372kB slab_unreclaimable:2032140kB kernel_stack:4736kB pagetables:5604kB unstable:0kB bounce:0kB free_pcp:1788kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:33:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:33:11 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:33:11 oak-gw06 kernel: Node 0 DMA32: 6388*4kB (UEM) 14080*8kB (UEM) 14281*16kB (UEM) 5579*32kB (UEM) 1054*64kB (UM) 26*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 616000kB Aug 15 01:33:11 oak-gw06 kernel: Node 0 Normal: 21489*4kB (UE) 76160*8kB (UEM) 59470*16kB (UEM) 12444*32kB (UEM) 857*64kB (UM) 32*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2103908kB Aug 15 01:33:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:33:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:33:11 oak-gw06 kernel: 2084986 total pagecache pages Aug 15 01:33:11 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:33:11 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:33:11 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:33:11 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:33:11 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:33:11 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:33:11 oak-gw06 kernel: 127313 pages reserved Aug 15 01:33:11 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 01:33:11 oak-gw06 kernel: CPU: 5 PID: 5330 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:33:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:33:11 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:33:11 oak-gw06 kernel: 00000000000080d0 000000007e5cf650 ffff8800afe77808 ffffffff8168662f Aug 15 01:33:11 oak-gw06 kernel: ffff8800afe77898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:33:11 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800afe77868 000000007e5cf650 Aug 15 01:33:11 oak-gw06 kernel: Call Trace: Aug 15 01:33:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:33:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:33:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:33:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:33:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 01:33:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 01:33:11 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 01:33:11 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 01:33:11 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:33:11 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:33:11 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:33:11 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:33:11 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:33:11 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:33:11 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:33:11 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:33:11 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:33:11 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:33:11 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:33:11 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:33:11 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:33:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:33:11 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:33:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:33:11 oak-gw06 kernel: Mem-Info: Aug 15 01:33:11 oak-gw06 kernel: active_anon:18218 inactive_anon:51094 isolated_anon:0#012 active_file:653605 inactive_file:1942689 isolated_file:0#012 unevictable:0 dirty:4386 writeback:2079 unstable:0#012 slab_reclaimable:32671 slab_unreclaimable:590821#012 mapped:10668 shmem:45078 pagetables:1669 bounce:0#012 free:683439 free_pcp:312 free_cma:0 Aug 15 01:33:11 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:33:11 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:33:11 oak-gw06 kernel: Node 0 DMA32 free:613984kB min:69724kB low:87152kB high:104584kB active_anon:9840kB inactive_anon:35584kB active_file:489808kB inactive_file:1339116kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2932kB writeback:448kB mapped:4824kB shmem:31268kB slab_reclaimable:18312kB slab_unreclaimable:331128kB kernel_stack:960kB pagetables:1072kB unstable:0kB bounce:0kB free_pcp:552kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:33:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:33:11 oak-gw06 kernel: Node 0 Normal free:2101492kB min:323104kB low:403880kB high:484656kB active_anon:64852kB inactive_anon:168792kB active_file:2135240kB inactive_file:6422052kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:14612kB writeback:884kB mapped:37848kB shmem:149044kB slab_reclaimable:112372kB slab_unreclaimable:2032140kB kernel_stack:4736kB pagetables:5604kB unstable:0kB bounce:0kB free_pcp:704kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:33:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:33:11 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:33:11 oak-gw06 kernel: Node 0 DMA32: 6388*4kB (UEM) 14080*8kB (UEM) 14281*16kB (UEM) 5579*32kB (UEM) 1054*64kB (UM) 26*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 616000kB Aug 15 01:33:11 oak-gw06 kernel: Node 0 Normal: 21746*4kB (UEM) 75491*8kB (UEM) 59492*16kB (UEM) 12447*32kB (UEM) 857*64kB (UM) 32*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2100032kB Aug 15 01:33:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:33:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:33:11 oak-gw06 kernel: 2086641 total pagecache pages Aug 15 01:33:11 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:33:11 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:33:11 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:33:11 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:33:11 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:33:11 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:33:11 oak-gw06 kernel: 127313 pages reserved Aug 15 01:38:11 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 01:38:11 oak-gw06 kernel: CPU: 6 PID: 5350 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:38:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:38:11 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:38:11 oak-gw06 kernel: 00000000000080d0 00000000ab426453 ffff8802412db858 ffffffff8168662f Aug 15 01:38:11 oak-gw06 kernel: ffff8802412db8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:38:11 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8802412db8b8 00000000ab426453 Aug 15 01:38:11 oak-gw06 kernel: Call Trace: Aug 15 01:38:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:38:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:38:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:38:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:38:11 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 01:38:11 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 01:38:11 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:38:11 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:38:11 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:38:11 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:38:11 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:38:11 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:38:11 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:38:11 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:38:11 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:38:11 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:38:11 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:38:11 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:38:11 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:38:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:38:11 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:38:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:38:11 oak-gw06 kernel: Mem-Info: Aug 15 01:38:11 oak-gw06 kernel: active_anon:24626 inactive_anon:51094 isolated_anon:0#012 active_file:703991 inactive_file:1472749 isolated_file:0#012 unevictable:0 dirty:6003 writeback:108 unstable:0#012 slab_reclaimable:32613 slab_unreclaimable:580794#012 mapped:10685 shmem:45078 pagetables:1702 bounce:0#012 free:1106985 free_pcp:220 free_cma:0 Aug 15 01:38:11 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:38:11 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:38:11 oak-gw06 kernel: Node 0 DMA32 free:870860kB min:69724kB low:87152kB high:104584kB active_anon:14640kB inactive_anon:35584kB active_file:521664kB inactive_file:1041948kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:5996kB writeback:0kB mapped:4828kB shmem:31268kB slab_reclaimable:18292kB slab_unreclaimable:327244kB kernel_stack:944kB pagetables:1076kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:38:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:38:11 oak-gw06 kernel: Node 0 Normal free:3539932kB min:323104kB low:403880kB high:484656kB active_anon:83864kB inactive_anon:168792kB active_file:2302360kB inactive_file:4840988kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:17240kB writeback:1208kB mapped:37912kB shmem:149044kB slab_reclaimable:112160kB slab_unreclaimable:1995916kB kernel_stack:4752kB pagetables:5732kB unstable:0kB bounce:0kB free_pcp:1728kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:38:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:38:11 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:38:11 oak-gw06 kernel: Node 0 DMA32: 5813*4kB (UEM) 19580*8kB (UEM) 16599*16kB (UEM) 8896*32kB (UEM) 1896*64kB (UM) 157*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 871844kB Aug 15 01:38:11 oak-gw06 kernel: Node 0 Normal: 32497*4kB (UEM) 111688*8kB (UEM) 74310*16kB (UEM) 32052*32kB (UEM) 4617*64kB (UEM) 52*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3540516kB Aug 15 01:38:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:38:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:38:11 oak-gw06 kernel: 2067573 total pagecache pages Aug 15 01:38:11 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:38:11 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:38:11 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:38:11 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:38:11 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:38:11 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:38:11 oak-gw06 kernel: 127313 pages reserved Aug 15 01:38:11 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 01:38:11 oak-gw06 kernel: CPU: 6 PID: 5350 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:38:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:38:11 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:38:11 oak-gw06 kernel: 00000000000080d0 00000000ab426453 ffff8802412db808 ffffffff8168662f Aug 15 01:38:11 oak-gw06 kernel: ffff8802412db898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 01:38:11 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8802412db868 00000000ab426453 Aug 15 01:38:11 oak-gw06 kernel: Call Trace: Aug 15 01:38:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:38:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:38:11 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 01:38:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:38:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:38:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 01:38:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 01:38:11 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 01:38:11 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 01:38:11 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:38:11 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:38:11 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:38:11 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:38:11 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:38:11 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:38:11 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:38:11 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:38:11 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:38:11 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:38:11 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:38:11 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:38:11 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:38:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:38:11 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:38:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:38:11 oak-gw06 kernel: Mem-Info: Aug 15 01:38:11 oak-gw06 kernel: active_anon:24626 inactive_anon:51094 isolated_anon:0#012 active_file:709679 inactive_file:1466996 isolated_file:0#012 unevictable:0 dirty:5809 writeback:108 unstable:0#012 slab_reclaimable:32613 slab_unreclaimable:580794#012 mapped:10685 shmem:45078 pagetables:1702 bounce:0#012 free:1106996 free_pcp:189 free_cma:0 Aug 15 01:38:11 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:38:11 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:38:11 oak-gw06 kernel: Node 0 DMA32 free:870860kB min:69724kB low:87152kB high:104584kB active_anon:14640kB inactive_anon:35584kB active_file:525696kB inactive_file:1037916kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:5996kB writeback:0kB mapped:4828kB shmem:31268kB slab_reclaimable:18292kB slab_unreclaimable:327244kB kernel_stack:944kB pagetables:1076kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:38:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:38:11 oak-gw06 kernel: Node 0 Normal free:3540960kB min:323104kB low:403880kB high:484656kB active_anon:84124kB inactive_anon:168792kB active_file:2313020kB inactive_file:4830068kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:17240kB writeback:432kB mapped:37912kB shmem:149044kB slab_reclaimable:112160kB slab_unreclaimable:1995916kB kernel_stack:4752kB pagetables:5732kB unstable:0kB bounce:0kB free_pcp:880kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:38:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:38:11 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:38:11 oak-gw06 kernel: Node 0 DMA32: 5813*4kB (UEM) 19580*8kB (UEM) 16599*16kB (UEM) 8896*32kB (UEM) 1896*64kB (UM) 157*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 871844kB Aug 15 01:38:11 oak-gw06 kernel: Node 0 Normal: 32588*4kB (UEM) 111750*8kB (UEM) 74310*16kB (UEM) 32052*32kB (UEM) 4617*64kB (UEM) 52*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3541376kB Aug 15 01:38:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:38:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:38:11 oak-gw06 kernel: 2067476 total pagecache pages Aug 15 01:38:11 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:38:11 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:38:11 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:38:11 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:38:11 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:38:11 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:38:11 oak-gw06 kernel: 127313 pages reserved Aug 15 01:43:11 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 15 01:43:11 oak-gw06 kernel: CPU: 6 PID: 5392 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:43:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:43:11 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:43:11 oak-gw06 kernel: 00000000000080d0 0000000095ef2ad1 ffff88010c8bf858 ffffffff8168662f Aug 15 01:43:11 oak-gw06 kernel: ffff88010c8bf8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:43:11 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88010c8bf8b8 0000000095ef2ad1 Aug 15 01:43:11 oak-gw06 kernel: Call Trace: Aug 15 01:43:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:43:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:43:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:43:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:43:11 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 01:43:11 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 01:43:11 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:43:11 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:43:11 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:43:11 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:43:11 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:43:11 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:43:11 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:43:11 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:43:11 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:43:11 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:43:11 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:43:11 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:43:11 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:43:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:43:11 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:43:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:43:11 oak-gw06 kernel: Mem-Info: Aug 15 01:43:11 oak-gw06 kernel: active_anon:24612 inactive_anon:51094 isolated_anon:0#012 active_file:2008429 inactive_file:72877 isolated_file:0#012 unevictable:0 dirty:9894 writeback:666 unstable:0#012 slab_reclaimable:32598 slab_unreclaimable:576327#012 mapped:10697 shmem:45078 pagetables:1689 bounce:0#012 free:1206752 free_pcp:349 free_cma:0 Aug 15 01:43:11 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:43:11 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:43:11 oak-gw06 kernel: Node 0 DMA32 free:1401784kB min:69724kB low:87152kB high:104584kB active_anon:11888kB inactive_anon:35584kB active_file:1000324kB inactive_file:44528kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:6264kB writeback:0kB mapped:4832kB shmem:31268kB slab_reclaimable:18288kB slab_unreclaimable:320816kB kernel_stack:944kB pagetables:1064kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:43:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:43:11 oak-gw06 kernel: Node 0 Normal free:3407720kB min:323104kB low:403880kB high:484656kB active_anon:86560kB inactive_anon:168792kB active_file:7033392kB inactive_file:248280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:32536kB writeback:4604kB mapped:37956kB shmem:149044kB slab_reclaimable:112104kB slab_unreclaimable:1984476kB kernel_stack:4768kB pagetables:5692kB unstable:0kB bounce:0kB free_pcp:1988kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:43:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:43:11 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:43:11 oak-gw06 kernel: Node 0 DMA32: 7190*4kB (UEM) 6338*8kB (UEM) 16871*16kB (UEM) 17071*32kB (UEM) 6082*64kB (UEM) 867*128kB (UM) 30*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1403576kB Aug 15 01:43:11 oak-gw06 kernel: Node 0 Normal: 35023*4kB (UEM) 41733*8kB (UEM) 84137*16kB (UEM) 38018*32kB (UEM) 5389*64kB (UEM) 200*128kB (UM) 2*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3407732kB Aug 15 01:43:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:43:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:43:11 oak-gw06 kernel: 2099649 total pagecache pages Aug 15 01:43:11 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:43:11 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:43:11 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:43:11 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:43:11 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:43:11 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:43:11 oak-gw06 kernel: 127313 pages reserved Aug 15 01:43:11 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 15 01:43:11 oak-gw06 kernel: CPU: 6 PID: 5392 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:43:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:43:11 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:43:11 oak-gw06 kernel: 00000000000080d0 0000000095ef2ad1 ffff88010c8bf808 ffffffff8168662f Aug 15 01:43:11 oak-gw06 kernel: ffff88010c8bf898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:43:11 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88010c8bf868 0000000095ef2ad1 Aug 15 01:43:11 oak-gw06 kernel: Call Trace: Aug 15 01:43:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:43:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:43:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:43:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:43:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 01:43:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 01:43:11 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 01:43:11 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 01:43:11 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:43:11 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:43:11 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:43:11 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:43:11 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:43:11 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:43:11 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:43:11 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:43:11 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:43:11 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:43:11 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:43:11 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:43:11 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:43:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:43:11 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:43:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:43:11 oak-gw06 kernel: Mem-Info: Aug 15 01:43:11 oak-gw06 kernel: active_anon:24612 inactive_anon:51094 isolated_anon:0#012 active_file:2008364 inactive_file:74957 isolated_file:0#012 unevictable:0 dirty:9991 writeback:569 unstable:0#012 slab_reclaimable:32598 slab_unreclaimable:576327#012 mapped:10697 shmem:45078 pagetables:1689 bounce:0#012 free:1204609 free_pcp:675 free_cma:0 Aug 15 01:43:11 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:43:11 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:43:11 oak-gw06 kernel: Node 0 DMA32 free:1401784kB min:69724kB low:87152kB high:104584kB active_anon:11888kB inactive_anon:35584kB active_file:1000324kB inactive_file:44528kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:6264kB writeback:0kB mapped:4832kB shmem:31268kB slab_reclaimable:18288kB slab_unreclaimable:320816kB kernel_stack:944kB pagetables:1064kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:43:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:43:11 oak-gw06 kernel: Node 0 Normal free:3399244kB min:323104kB low:403880kB high:484656kB active_anon:86560kB inactive_anon:168792kB active_file:7033132kB inactive_file:256340kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:33700kB writeback:1112kB mapped:37956kB shmem:149044kB slab_reclaimable:112104kB slab_unreclaimable:1984476kB kernel_stack:4768kB pagetables:5692kB unstable:0kB bounce:0kB free_pcp:3000kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:43:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:43:11 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:43:11 oak-gw06 kernel: Node 0 DMA32: 7190*4kB (UEM) 6338*8kB (UEM) 16871*16kB (UEM) 17071*32kB (UEM) 6082*64kB (UEM) 867*128kB (UM) 30*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1403576kB Aug 15 01:43:11 oak-gw06 kernel: Node 0 Normal: 34516*4kB (UEM) 40678*8kB (UEM) 84133*16kB (UEM) 38020*32kB (UEM) 5389*64kB (UEM) 200*128kB (UM) 2*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3397264kB Aug 15 01:43:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:43:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:43:11 oak-gw06 kernel: 2101783 total pagecache pages Aug 15 01:43:11 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:43:11 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:43:11 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:43:11 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:43:11 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:43:11 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:43:11 oak-gw06 kernel: 127313 pages reserved Aug 15 01:48:12 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 01:48:12 oak-gw06 kernel: CPU: 6 PID: 5330 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:48:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:48:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:48:12 oak-gw06 kernel: 00000000000080d0 000000007e5cf650 ffff8800afe77858 ffffffff8168662f Aug 15 01:48:12 oak-gw06 kernel: ffff8800afe778e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 15 01:48:12 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff8800afe778e8 000000007e5cf650 Aug 15 01:48:12 oak-gw06 kernel: Call Trace: Aug 15 01:48:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:48:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:48:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:48:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:48:12 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 01:48:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 01:48:12 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:48:12 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:48:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:48:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:48:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:48:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:48:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:48:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:48:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:48:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:48:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:48:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:48:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:48:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:48:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:48:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:48:12 oak-gw06 kernel: Mem-Info: Aug 15 01:48:12 oak-gw06 kernel: active_anon:24424 inactive_anon:51094 isolated_anon:0#012 active_file:2004889 inactive_file:6586 isolated_file:23#012 unevictable:0 dirty:17435 writeback:8853 unstable:0#012 slab_reclaimable:32592 slab_unreclaimable:571584#012 mapped:10708 shmem:45078 pagetables:1684 bounce:0#012 free:1247488 free_pcp:1304 free_cma:0 Aug 15 01:48:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:48:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:48:12 oak-gw06 kernel: Node 0 DMA32 free:1359240kB min:69724kB low:87152kB high:104584kB active_anon:11892kB inactive_anon:35584kB active_file:1071856kB inactive_file:572kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:10160kB writeback:4kB mapped:4832kB shmem:31268kB slab_reclaimable:18280kB slab_unreclaimable:317492kB kernel_stack:944kB pagetables:1080kB unstable:0kB bounce:0kB free_pcp:1756kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:48:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:48:12 oak-gw06 kernel: Node 0 Normal free:3619444kB min:323104kB low:403880kB high:484656kB active_anon:86064kB inactive_anon:168792kB active_file:6950108kB inactive_file:40524kB unevictable:0kB isolated(anon):0kB isolated(file):92kB present:13631488kB managed:13367060kB mlocked:0kB dirty:61132kB writeback:50928kB mapped:38000kB shmem:149044kB slab_reclaimable:112088kB slab_unreclaimable:1968828kB kernel_stack:4752kB pagetables:5656kB unstable:0kB bounce:0kB free_pcp:3944kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:48:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:48:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:48:12 oak-gw06 kernel: Node 0 DMA32: 4755*4kB (UEM) 10384*8kB (UEM) 9343*16kB (UEM) 16454*32kB (UEM) 6646*64kB (UEM) 1151*128kB (UM) 47*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1362812kB Aug 15 01:48:12 oak-gw06 kernel: Node 0 Normal: 26626*4kB (UE) 79704*8kB (UEM) 81259*16kB (UEM) 37246*32kB (UEM) 5523*64kB (UEM) 239*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3620984kB Aug 15 01:48:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:48:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:48:12 oak-gw06 kernel: 2058108 total pagecache pages Aug 15 01:48:12 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:48:12 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:48:12 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:48:12 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:48:12 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:48:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:48:12 oak-gw06 kernel: 127313 pages reserved Aug 15 01:48:12 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 01:48:12 oak-gw06 kernel: CPU: 6 PID: 5330 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:48:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:48:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:48:12 oak-gw06 kernel: 00000000000080d0 000000007e5cf650 ffff8800afe77808 ffffffff8168662f Aug 15 01:48:12 oak-gw06 kernel: ffff8800afe77898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 01:48:12 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800afe77868 000000007e5cf650 Aug 15 01:48:12 oak-gw06 kernel: Call Trace: Aug 15 01:48:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:48:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:48:12 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 01:48:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:48:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:48:12 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 01:48:12 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 01:48:12 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 01:48:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 01:48:12 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:48:12 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:48:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:48:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:48:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:48:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:48:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:48:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:48:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:48:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:48:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:48:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:48:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:48:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:48:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:48:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:48:12 oak-gw06 kernel: Mem-Info: Aug 15 01:48:12 oak-gw06 kernel: active_anon:24424 inactive_anon:51094 isolated_anon:0#012 active_file:2005426 inactive_file:22429 isolated_file:23#012 unevictable:0 dirty:17144 writeback:10987 unstable:0#012 slab_reclaimable:32592 slab_unreclaimable:571584#012 mapped:10708 shmem:45078 pagetables:1684 bounce:0#012 free:1254357 free_pcp:685 free_cma:0 Aug 15 01:48:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:48:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:48:12 oak-gw06 kernel: Node 0 DMA32 free:1370804kB min:69724kB low:87152kB high:104584kB active_anon:11892kB inactive_anon:35584kB active_file:1071856kB inactive_file:572kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:10160kB writeback:4kB mapped:4832kB shmem:31268kB slab_reclaimable:18280kB slab_unreclaimable:317492kB kernel_stack:944kB pagetables:1080kB unstable:0kB bounce:0kB free_pcp:1356kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:48:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:48:12 oak-gw06 kernel: Node 0 Normal free:3631676kB min:323104kB low:403880kB high:484656kB active_anon:85804kB inactive_anon:168792kB active_file:6949848kB inactive_file:103444kB unevictable:0kB isolated(anon):0kB isolated(file):92kB present:13631488kB managed:13367060kB mlocked:0kB dirty:51044kB writeback:42004kB mapped:38000kB shmem:149044kB slab_reclaimable:112088kB slab_unreclaimable:1968828kB kernel_stack:4752kB pagetables:5656kB unstable:0kB bounce:0kB free_pcp:2256kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:48:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:48:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:48:12 oak-gw06 kernel: Node 0 DMA32: 7506*4kB (UEM) 10397*8kB (UEM) 9590*16kB (UEM) 16455*32kB (UEM) 6647*64kB (UEM) 1151*128kB (UM) 47*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1377968kB Aug 15 01:48:12 oak-gw06 kernel: Node 0 Normal: 34128*4kB (UE) 72309*8kB (UEM) 83770*16kB (UEM) 37261*32kB (UEM) 5523*64kB (UEM) 239*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3632488kB Aug 15 01:48:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:48:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:48:12 oak-gw06 kernel: 2072600 total pagecache pages Aug 15 01:48:12 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:48:12 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:48:12 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:48:12 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:48:12 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:48:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:48:12 oak-gw06 kernel: 127313 pages reserved Aug 15 01:53:11 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 15 01:53:11 oak-gw06 kernel: CPU: 6 PID: 5392 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:53:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:53:11 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:53:11 oak-gw06 kernel: 00000000000080d0 0000000095ef2ad1 ffff88010c8bf858 ffffffff8168662f Aug 15 01:53:11 oak-gw06 kernel: ffff88010c8bf8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:53:11 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88010c8bf8b8 0000000095ef2ad1 Aug 15 01:53:11 oak-gw06 kernel: Call Trace: Aug 15 01:53:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:53:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:53:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:53:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:53:11 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 01:53:11 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 01:53:11 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:53:11 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:53:11 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:53:11 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:53:11 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:53:11 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:53:11 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:53:11 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:53:11 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:53:11 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:53:11 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:53:11 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:53:11 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:53:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:53:11 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:53:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:53:11 oak-gw06 kernel: Mem-Info: Aug 15 01:53:11 oak-gw06 kernel: active_anon:25389 inactive_anon:51094 isolated_anon:0#012 active_file:1988237 inactive_file:30891 isolated_file:0#012 unevictable:0 dirty:9701 writeback:225 unstable:0#012 slab_reclaimable:32591 slab_unreclaimable:570368#012 mapped:10715 shmem:45078 pagetables:1694 bounce:0#012 free:1205206 free_pcp:1595 free_cma:0 Aug 15 01:53:11 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:53:11 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:53:11 oak-gw06 kernel: Node 0 DMA32 free:1190820kB min:69724kB low:87152kB high:104584kB active_anon:10804kB inactive_anon:35584kB active_file:1193460kB inactive_file:17684kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:6440kB writeback:124kB mapped:4840kB shmem:31268kB slab_reclaimable:18280kB slab_unreclaimable:322920kB kernel_stack:944kB pagetables:1072kB unstable:0kB bounce:0kB free_pcp:3616kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:53:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:53:11 oak-gw06 kernel: Node 0 Normal free:3624888kB min:323104kB low:403880kB high:484656kB active_anon:90752kB inactive_anon:168792kB active_file:6742456kB inactive_file:101460kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:31976kB writeback:1940kB mapped:38020kB shmem:149044kB slab_reclaimable:112084kB slab_unreclaimable:1958536kB kernel_stack:4752kB pagetables:5704kB unstable:0kB bounce:0kB free_pcp:3948kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:53:11 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:53:11 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:53:11 oak-gw06 kernel: Node 0 DMA32: 7355*4kB (UEM) 7507*8kB (UEM) 1088*16kB (UEM) 14734*32kB (UEM) 6776*64kB (UEM) 1297*128kB (UM) 61*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1193668kB Aug 15 01:53:11 oak-gw06 kernel: Node 0 Normal: 26485*4kB (UEM) 81038*8kB (UEM) 81803*16kB (UEM) 36137*32kB (UEM) 5747*64kB (UEM) 284*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3624404kB Aug 15 01:53:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:53:11 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:53:11 oak-gw06 kernel: 2053500 total pagecache pages Aug 15 01:53:11 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:53:11 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:53:11 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:53:11 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:53:11 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:53:11 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:53:11 oak-gw06 kernel: 127313 pages reserved Aug 15 01:53:11 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 15 01:53:11 oak-gw06 kernel: CPU: 6 PID: 5392 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:53:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:53:11 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:53:11 oak-gw06 kernel: 00000000000080d0 0000000095ef2ad1 ffff88010c8bf808 ffffffff8168662f Aug 15 01:53:11 oak-gw06 kernel: ffff88010c8bf898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 01:53:11 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88010c8bf868 0000000095ef2ad1 Aug 15 01:53:11 oak-gw06 kernel: Call Trace: Aug 15 01:53:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:53:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:53:11 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 01:53:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:53:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:53:11 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 01:53:11 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 01:53:11 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 01:53:11 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 01:53:11 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:53:11 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:53:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:53:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:53:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:53:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:53:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:53:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:53:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:53:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:53:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:53:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:53:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:53:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:53:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:53:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:53:12 oak-gw06 kernel: Mem-Info: Aug 15 01:53:12 oak-gw06 kernel: active_anon:25389 inactive_anon:51094 isolated_anon:0#012 active_file:1981180 inactive_file:27799 isolated_file:0#012 unevictable:0 dirty:9604 writeback:831 unstable:0#012 slab_reclaimable:32591 slab_unreclaimable:570300#012 mapped:10715 shmem:45078 pagetables:1694 bounce:0#012 free:1213141 free_pcp:1151 free_cma:0 Aug 15 01:53:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:53:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:53:12 oak-gw06 kernel: Node 0 DMA32 free:1207996kB min:69724kB low:87152kB high:104584kB active_anon:10804kB inactive_anon:35584kB active_file:1190940kB inactive_file:14156kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:6440kB writeback:0kB mapped:4840kB shmem:31268kB slab_reclaimable:18280kB slab_unreclaimable:322920kB kernel_stack:944kB pagetables:1072kB unstable:0kB bounce:0kB free_pcp:1324kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:53:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:53:12 oak-gw06 kernel: Node 0 Normal free:3636568kB min:323104kB low:403880kB high:484656kB active_anon:91792kB inactive_anon:168792kB active_file:6734916kB inactive_file:95220kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:32364kB writeback:4268kB mapped:38020kB shmem:149044kB slab_reclaimable:112084kB slab_unreclaimable:1958264kB kernel_stack:4752kB pagetables:5704kB unstable:0kB bounce:0kB free_pcp:4044kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:53:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:53:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:53:12 oak-gw06 kernel: Node 0 DMA32: 9572*4kB (UEM) 7819*8kB (UEM) 1608*16kB (UEM) 14760*32kB (UEM) 6776*64kB (UEM) 1297*128kB (UM) 61*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1214184kB Aug 15 01:53:12 oak-gw06 kernel: Node 0 Normal: 28914*4kB (UEM) 81214*8kB (UEM) 82138*16kB (UEM) 36148*32kB (UEM) 5748*64kB (UEM) 284*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3641304kB Aug 15 01:53:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:53:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:53:12 oak-gw06 kernel: 2051038 total pagecache pages Aug 15 01:53:12 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:53:12 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:53:12 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:53:12 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:53:12 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:53:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:53:12 oak-gw06 kernel: 127313 pages reserved Aug 15 01:58:11 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 01:58:11 oak-gw06 kernel: CPU: 6 PID: 5422 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:58:11 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:58:11 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:58:11 oak-gw06 kernel: 00000000000080d0 0000000021355c44 ffff88024b77f858 ffffffff8168662f Aug 15 01:58:11 oak-gw06 kernel: ffff88024b77f8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 01:58:11 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88024b77f8b8 0000000021355c44 Aug 15 01:58:11 oak-gw06 kernel: Call Trace: Aug 15 01:58:11 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:58:11 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:58:11 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:58:11 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:58:11 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 01:58:11 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 01:58:11 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:58:11 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:58:11 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:58:11 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:58:11 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:58:11 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:58:11 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:58:11 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:58:11 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:58:11 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:58:11 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:58:11 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:58:11 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:58:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:58:11 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:58:11 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:58:11 oak-gw06 kernel: Mem-Info: Aug 15 01:58:11 oak-gw06 kernel: active_anon:24621 inactive_anon:51094 isolated_anon:0#012 active_file:2031817 inactive_file:28574 isolated_file:0#012 unevictable:0 dirty:10170 writeback:1873 unstable:0#012 slab_reclaimable:32590 slab_unreclaimable:569717#012 mapped:10730 shmem:45078 pagetables:1692 bounce:0#012 free:1233772 free_pcp:777 free_cma:0 Aug 15 01:58:11 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:58:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:58:12 oak-gw06 kernel: Node 0 DMA32 free:1283200kB min:69724kB low:87152kB high:104584kB active_anon:13840kB inactive_anon:35584kB active_file:1142488kB inactive_file:21392kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:6292kB writeback:292kB mapped:4832kB shmem:31268kB slab_reclaimable:18280kB slab_unreclaimable:318960kB kernel_stack:928kB pagetables:1072kB unstable:0kB bounce:0kB free_pcp:388kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:58:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:58:12 oak-gw06 kernel: Node 0 Normal free:3632488kB min:323104kB low:403880kB high:484656kB active_anon:84904kB inactive_anon:168792kB active_file:6984780kB inactive_file:96544kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:35552kB writeback:2156kB mapped:38088kB shmem:149044kB slab_reclaimable:112080kB slab_unreclaimable:1959892kB kernel_stack:4784kB pagetables:5696kB unstable:0kB bounce:0kB free_pcp:2952kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:58:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:58:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:58:12 oak-gw06 kernel: Node 0 DMA32: 7016*4kB (UEM) 7885*8kB (UEM) 2402*16kB (UEM) 15721*32kB (UEM) 6921*64kB (UEM) 1464*128kB (UM) 88*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1285512kB Aug 15 01:58:12 oak-gw06 kernel: Node 0 Normal: 36535*4kB (UEM) 63027*8kB (UEM) 83010*16kB (UEM) 37520*32kB (UEM) 6277*64kB (UEM) 355*128kB (UM) 5*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3627604kB Aug 15 01:58:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:58:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:58:12 oak-gw06 kernel: 2089281 total pagecache pages Aug 15 01:58:12 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:58:12 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:58:12 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:58:12 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:58:12 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:58:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:58:12 oak-gw06 kernel: 127313 pages reserved Aug 15 01:58:12 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 01:58:12 oak-gw06 kernel: CPU: 6 PID: 5422 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 01:58:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 01:58:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 01:58:12 oak-gw06 kernel: 00000000000080d0 0000000021355c44 ffff88024b77f808 ffffffff8168662f Aug 15 01:58:12 oak-gw06 kernel: ffff88024b77f898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 01:58:12 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88024b77f868 0000000021355c44 Aug 15 01:58:12 oak-gw06 kernel: Call Trace: Aug 15 01:58:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 01:58:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 01:58:12 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 01:58:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 01:58:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 01:58:12 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 01:58:12 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 01:58:12 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 01:58:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 01:58:12 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 01:58:12 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 01:58:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 01:58:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 01:58:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 01:58:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 01:58:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 01:58:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 01:58:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 01:58:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 01:58:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 01:58:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 01:58:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 01:58:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:58:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 01:58:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 01:58:12 oak-gw06 kernel: Mem-Info: Aug 15 01:58:12 oak-gw06 kernel: active_anon:24621 inactive_anon:51094 isolated_anon:0#012 active_file:2031687 inactive_file:31629 isolated_file:0#012 unevictable:0 dirty:10461 writeback:30 unstable:0#012 slab_reclaimable:32590 slab_unreclaimable:569717#012 mapped:10730 shmem:45078 pagetables:1692 bounce:0#012 free:1231193 free_pcp:416 free_cma:0 Aug 15 01:58:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 01:58:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 01:58:12 oak-gw06 kernel: Node 0 DMA32 free:1284728kB min:69724kB low:87152kB high:104584kB active_anon:13840kB inactive_anon:35584kB active_file:1142488kB inactive_file:21392kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:6292kB writeback:292kB mapped:4832kB shmem:31268kB slab_reclaimable:18280kB slab_unreclaimable:318960kB kernel_stack:928kB pagetables:1072kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:58:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 01:58:12 oak-gw06 kernel: Node 0 Normal free:3620588kB min:323104kB low:403880kB high:484656kB active_anon:84644kB inactive_anon:168792kB active_file:6984260kB inactive_file:108052kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:35620kB writeback:1876kB mapped:38088kB shmem:149044kB slab_reclaimable:112080kB slab_unreclaimable:1959852kB kernel_stack:4784kB pagetables:5696kB unstable:0kB bounce:0kB free_pcp:2648kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 01:58:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 01:58:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 01:58:12 oak-gw06 kernel: Node 0 DMA32: 7113*4kB (UEM) 7887*8kB (UEM) 2434*16kB (UEM) 15721*32kB (UEM) 6921*64kB (UEM) 1464*128kB (UM) 88*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1286428kB Aug 15 01:58:12 oak-gw06 kernel: Node 0 Normal: 35209*4kB (UE) 62700*8kB (UEM) 83019*16kB (UEM) 37520*32kB (UEM) 6277*64kB (UEM) 355*128kB (UM) 5*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3619828kB Aug 15 01:58:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 01:58:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 01:58:12 oak-gw06 kernel: 2091648 total pagecache pages Aug 15 01:58:12 oak-gw06 kernel: 16 pages in swap cache Aug 15 01:58:12 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 01:58:12 oak-gw06 kernel: Free swap = 4194036kB Aug 15 01:58:12 oak-gw06 kernel: Total swap = 4194300kB Aug 15 01:58:12 oak-gw06 kernel: 4194203 pages RAM Aug 15 01:58:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 01:58:12 oak-gw06 kernel: 127313 pages reserved Aug 15 02:03:12 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 02:03:12 oak-gw06 kernel: CPU: 6 PID: 5433 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 02:03:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 02:03:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 02:03:12 oak-gw06 kernel: 00000000000080d0 000000000f3da7d5 ffff88039a6c7858 ffffffff8168662f Aug 15 02:03:12 oak-gw06 kernel: ffff88039a6c78e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 02:03:12 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88039a6c78b8 000000000f3da7d5 Aug 15 02:03:12 oak-gw06 kernel: Call Trace: Aug 15 02:03:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 02:03:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 02:03:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 02:03:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 02:03:12 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 02:03:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 02:03:12 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 02:03:12 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 02:03:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 02:03:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 02:03:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 02:03:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 02:03:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 02:03:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 02:03:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 02:03:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 02:03:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 02:03:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 02:03:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 02:03:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 02:03:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 02:03:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 02:03:12 oak-gw06 kernel: Mem-Info: Aug 15 02:03:12 oak-gw06 kernel: active_anon:12619 inactive_anon:51094 isolated_anon:0#012 active_file:2050163 inactive_file:9562 isolated_file:0#012 unevictable:0 dirty:12180 writeback:164 unstable:0#012 slab_reclaimable:32590 slab_unreclaimable:568554#012 mapped:10490 shmem:45078 pagetables:1408 bounce:0#012 free:1248597 free_pcp:187 free_cma:0 Aug 15 02:03:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 02:03:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 02:03:12 oak-gw06 kernel: Node 0 DMA32 free:1082992kB min:69724kB low:87152kB high:104584kB active_anon:9816kB inactive_anon:35584kB active_file:1356580kB inactive_file:9792kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:6876kB writeback:0kB mapped:4724kB shmem:31268kB slab_reclaimable:18280kB slab_unreclaimable:319956kB kernel_stack:928kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 02:03:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 02:03:12 oak-gw06 kernel: Node 0 Normal free:3894848kB min:323104kB low:403880kB high:484656kB active_anon:40660kB inactive_anon:168792kB active_file:6844072kB inactive_file:28456kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:41844kB writeback:656kB mapped:37236kB shmem:149044kB slab_reclaimable:112080kB slab_unreclaimable:1954244kB kernel_stack:4768kB pagetables:4572kB unstable:0kB bounce:0kB free_pcp:1248kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 02:03:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 02:03:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 02:03:12 oak-gw06 kernel: Node 0 DMA32: 7693*4kB (UEM) 6793*8kB (UEM) 2080*16kB (UEM) 9113*32kB (UEM) 6910*64kB (UEM) 1580*128kB (UM) 117*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1084444kB Aug 15 02:03:12 oak-gw06 kernel: Node 0 Normal: 53882*4kB (UEM) 77467*8kB (UEM) 83395*16kB (UEM) 38069*32kB (UEM) 6967*64kB (UEM) 473*128kB (UM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3896272kB Aug 15 02:03:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 02:03:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 02:03:12 oak-gw06 kernel: 2074545 total pagecache pages Aug 15 02:03:12 oak-gw06 kernel: 16 pages in swap cache Aug 15 02:03:12 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 02:03:12 oak-gw06 kernel: Free swap = 4194036kB Aug 15 02:03:12 oak-gw06 kernel: Total swap = 4194300kB Aug 15 02:03:12 oak-gw06 kernel: 4194203 pages RAM Aug 15 02:03:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 02:03:12 oak-gw06 kernel: 127313 pages reserved Aug 15 02:03:12 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 02:03:12 oak-gw06 kernel: CPU: 1 PID: 5433 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 02:03:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 02:03:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 02:03:12 oak-gw06 kernel: 00000000000080d0 000000000f3da7d5 ffff88039a6c7808 ffffffff8168662f Aug 15 02:03:12 oak-gw06 kernel: ffff88039a6c7898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 02:03:12 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88039a6c7868 000000000f3da7d5 Aug 15 02:03:12 oak-gw06 kernel: Call Trace: Aug 15 02:03:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 02:03:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 02:03:12 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 02:03:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 02:03:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 02:03:12 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 02:03:12 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 02:03:12 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 02:03:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 02:03:12 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 02:03:12 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 02:03:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 02:03:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 02:03:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 02:03:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 02:03:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 02:03:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 02:03:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 02:03:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 02:03:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 02:03:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 02:03:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 02:03:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 02:03:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 02:03:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 02:03:12 oak-gw06 kernel: Mem-Info: Aug 15 02:03:12 oak-gw06 kernel: active_anon:12619 inactive_anon:51094 isolated_anon:0#012 active_file:2050033 inactive_file:9563 isolated_file:0#012 unevictable:0 dirty:12180 writeback:164 unstable:0#012 slab_reclaimable:32590 slab_unreclaimable:568554#012 mapped:10495 shmem:45078 pagetables:1408 bounce:0#012 free:1249018 free_pcp:190 free_cma:0 Aug 15 02:03:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 02:03:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 02:03:12 oak-gw06 kernel: Node 0 DMA32 free:1083564kB min:69724kB low:87152kB high:104584kB active_anon:9816kB inactive_anon:35584kB active_file:1356580kB inactive_file:9792kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:6876kB writeback:0kB mapped:4728kB shmem:31268kB slab_reclaimable:18280kB slab_unreclaimable:319956kB kernel_stack:928kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 02:03:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 02:03:12 oak-gw06 kernel: Node 0 Normal free:3895984kB min:323104kB low:403880kB high:484656kB active_anon:41180kB inactive_anon:168792kB active_file:6843308kB inactive_file:28492kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:41844kB writeback:656kB mapped:37260kB shmem:149044kB slab_reclaimable:112080kB slab_unreclaimable:1954244kB kernel_stack:4784kB pagetables:4572kB unstable:0kB bounce:0kB free_pcp:984kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 02:03:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 02:03:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 02:03:12 oak-gw06 kernel: Node 0 DMA32: 7693*4kB (UEM) 6793*8kB (UEM) 2080*16kB (UEM) 9113*32kB (UEM) 6910*64kB (UEM) 1580*128kB (UM) 117*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1084444kB Aug 15 02:03:12 oak-gw06 kernel: Node 0 Normal: 53975*4kB (UEM) 77468*8kB (UEM) 83394*16kB (UEM) 38069*32kB (UEM) 6967*64kB (UEM) 473*128kB (UM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3896636kB Aug 15 02:03:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 02:03:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 02:03:12 oak-gw06 kernel: 2074388 total pagecache pages Aug 15 02:03:12 oak-gw06 kernel: 16 pages in swap cache Aug 15 02:03:12 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 02:03:12 oak-gw06 kernel: Free swap = 4194036kB Aug 15 02:03:12 oak-gw06 kernel: Total swap = 4194300kB Aug 15 02:03:12 oak-gw06 kernel: 4194203 pages RAM Aug 15 02:03:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 02:03:12 oak-gw06 kernel: 127313 pages reserved Aug 15 02:08:12 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 15 02:08:12 oak-gw06 kernel: CPU: 0 PID: 5485 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 02:08:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 02:08:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 02:08:12 oak-gw06 kernel: 00000000000080d0 00000000433eacc0 ffff880210263858 ffffffff8168662f Aug 15 02:08:12 oak-gw06 kernel: ffff8802102638e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 02:08:12 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8802102638b8 00000000433eacc0 Aug 15 02:08:12 oak-gw06 kernel: Call Trace: Aug 15 02:08:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 02:08:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 02:08:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 02:08:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 02:08:12 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 02:08:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 02:08:12 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 02:08:12 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 02:08:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 02:08:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 02:08:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 02:08:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 02:08:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 02:08:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 02:08:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 02:08:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 02:08:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 02:08:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 02:08:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 02:08:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 02:08:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 02:08:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 02:08:12 oak-gw06 kernel: Mem-Info: Aug 15 02:08:12 oak-gw06 kernel: active_anon:15149 inactive_anon:51094 isolated_anon:0#012 active_file:2003324 inactive_file:4887 isolated_file:0#012 unevictable:0 dirty:7480 writeback:467 unstable:0#012 slab_reclaimable:32582 slab_unreclaimable:565900#012 mapped:10746 shmem:45078 pagetables:1631 bounce:0#012 free:1300264 free_pcp:189 free_cma:0 Aug 15 02:08:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 02:08:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 02:08:12 oak-gw06 kernel: Node 0 DMA32 free:1123252kB min:69724kB low:87152kB high:104584kB active_anon:9824kB inactive_anon:35584kB active_file:1324748kB inactive_file:1480kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:8716kB writeback:0kB mapped:4852kB shmem:31268kB slab_reclaimable:18272kB slab_unreclaimable:319424kB kernel_stack:928kB pagetables:1064kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 02:08:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 02:08:12 oak-gw06 kernel: Node 0 Normal free:4061436kB min:323104kB low:403880kB high:484656kB active_anon:50868kB inactive_anon:168792kB active_file:6696616kB inactive_file:9864kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:21272kB writeback:1140kB mapped:38160kB shmem:149044kB slab_reclaimable:112056kB slab_unreclaimable:1944224kB kernel_stack:4832kB pagetables:5468kB unstable:0kB bounce:0kB free_pcp:1376kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 02:08:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 02:08:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 02:08:12 oak-gw06 kernel: Node 0 DMA32: 7188*4kB (UEM) 6378*8kB (UEM) 2686*16kB (UEM) 9274*32kB (UEM) 7032*64kB (UEM) 1723*128kB (UM) 132*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1123904kB Aug 15 02:08:12 oak-gw06 kernel: Node 0 Normal: 36998*4kB (UEM) 99171*8kB (UEM) 82716*16kB (UEM) 38867*32kB (UEM) 7464*64kB (UEM) 571*128kB (UM) 9*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4061648kB Aug 15 02:08:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 02:08:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 02:08:12 oak-gw06 kernel: 2053323 total pagecache pages Aug 15 02:08:12 oak-gw06 kernel: 16 pages in swap cache Aug 15 02:08:12 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 02:08:12 oak-gw06 kernel: Free swap = 4194036kB Aug 15 02:08:12 oak-gw06 kernel: Total swap = 4194300kB Aug 15 02:08:12 oak-gw06 kernel: 4194203 pages RAM Aug 15 02:08:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 02:08:12 oak-gw06 kernel: 127313 pages reserved Aug 15 02:08:12 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 15 02:08:12 oak-gw06 kernel: CPU: 0 PID: 5485 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 02:08:12 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 02:08:12 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 02:08:12 oak-gw06 kernel: 00000000000080d0 00000000433eacc0 ffff880210263808 ffffffff8168662f Aug 15 02:08:12 oak-gw06 kernel: ffff880210263898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 02:08:12 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880210263868 00000000433eacc0 Aug 15 02:08:12 oak-gw06 kernel: Call Trace: Aug 15 02:08:12 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 02:08:12 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 02:08:12 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 02:08:12 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 02:08:12 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 02:08:12 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 02:08:12 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 02:08:12 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 02:08:12 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 02:08:12 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 02:08:12 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 02:08:12 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 02:08:12 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 02:08:12 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 02:08:12 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 02:08:12 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 02:08:12 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 02:08:12 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 02:08:12 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 02:08:12 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 02:08:12 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 02:08:12 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 02:08:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 02:08:12 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 02:08:12 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 02:08:12 oak-gw06 kernel: Mem-Info: Aug 15 02:08:12 oak-gw06 kernel: active_anon:15336 inactive_anon:51094 isolated_anon:0#012 active_file:2005908 inactive_file:2361 isolated_file:0#012 unevictable:0 dirty:7633 writeback:164 unstable:0#012 slab_reclaimable:32582 slab_unreclaimable:565934#012 mapped:10759 shmem:45078 pagetables:1650 bounce:0#012 free:1299749 free_pcp:212 free_cma:0 Aug 15 02:08:12 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 02:08:12 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 02:08:12 oak-gw06 kernel: Node 0 DMA32 free:1123904kB min:69724kB low:87152kB high:104584kB active_anon:9824kB inactive_anon:35584kB active_file:1324748kB inactive_file:1596kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:8664kB writeback:0kB mapped:4852kB shmem:31268kB slab_reclaimable:18272kB slab_unreclaimable:319424kB kernel_stack:928kB pagetables:1064kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 02:08:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 02:08:12 oak-gw06 kernel: Node 0 Normal free:4052476kB min:323104kB low:403880kB high:484656kB active_anon:52560kB inactive_anon:168792kB active_file:6698884kB inactive_file:9928kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:21868kB writeback:2596kB mapped:38184kB shmem:149044kB slab_reclaimable:112056kB slab_unreclaimable:1944296kB kernel_stack:4832kB pagetables:5536kB unstable:0kB bounce:0kB free_pcp:1804kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 02:08:12 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 02:08:12 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 02:08:12 oak-gw06 kernel: Node 0 DMA32: 7188*4kB (UEM) 6378*8kB (UEM) 2686*16kB (UEM) 9274*32kB (UEM) 7032*64kB (UEM) 1723*128kB (UM) 132*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1123904kB Aug 15 02:08:12 oak-gw06 kernel: Node 0 Normal: 35324*4kB (UEM) 98939*8kB (UEM) 82617*16kB (UEM) 38888*32kB (UEM) 7464*64kB (UEM) 571*128kB (UM) 9*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4052184kB Aug 15 02:08:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 02:08:12 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 02:08:12 oak-gw06 kernel: 2054915 total pagecache pages Aug 15 02:08:12 oak-gw06 kernel: 16 pages in swap cache Aug 15 02:08:12 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 02:08:12 oak-gw06 kernel: Free swap = 4194036kB Aug 15 02:08:12 oak-gw06 kernel: Total swap = 4194300kB Aug 15 02:08:12 oak-gw06 kernel: 4194203 pages RAM Aug 15 02:08:12 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 02:08:12 oak-gw06 kernel: 127313 pages reserved Aug 15 04:28:14 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 04:28:14 oak-gw06 kernel: CPU: 6 PID: 6356 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 04:28:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 04:28:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 04:28:14 oak-gw06 kernel: 00000000000080d0 000000001ab771ac ffff8803d77eb858 ffffffff8168662f Aug 15 04:28:14 oak-gw06 kernel: ffff8803d77eb8e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 15 04:28:14 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff8803d77eb8e8 000000001ab771ac Aug 15 04:28:14 oak-gw06 kernel: Call Trace: Aug 15 04:28:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 04:28:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 04:28:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 04:28:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 04:28:14 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 04:28:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 04:28:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 04:28:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 04:28:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 04:28:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 04:28:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 04:28:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 04:28:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 04:28:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 04:28:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 04:28:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 04:28:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 04:28:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 04:28:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 04:28:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:28:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 04:28:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:28:14 oak-gw06 kernel: Mem-Info: Aug 15 04:28:14 oak-gw06 kernel: active_anon:23441 inactive_anon:53142 isolated_anon:0#012 active_file:514653 inactive_file:2298967 isolated_file:0#012 unevictable:0 dirty:4686 writeback:1696 unstable:0#012 slab_reclaimable:32443 slab_unreclaimable:566913#012 mapped:10820 shmem:47126 pagetables:1693 bounce:0#012 free:482703 free_pcp:388 free_cma:0 Aug 15 04:28:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 04:28:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 04:28:14 oak-gw06 kernel: Node 0 DMA32 free:472504kB min:69724kB low:87152kB high:104584kB active_anon:11408kB inactive_anon:35584kB active_file:392048kB inactive_file:1599884kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3856kB writeback:0kB mapped:4872kB shmem:31268kB slab_reclaimable:18044kB slab_unreclaimable:304680kB kernel_stack:960kB pagetables:1276kB unstable:0kB bounce:0kB free_pcp:2996kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:28:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 04:28:14 oak-gw06 kernel: Node 0 Normal free:1449260kB min:323104kB low:403880kB high:484656kB active_anon:82356kB inactive_anon:176984kB active_file:1666564kB inactive_file:7583144kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:15276kB writeback:8192kB mapped:38408kB shmem:157236kB slab_reclaimable:111728kB slab_unreclaimable:1962956kB kernel_stack:4720kB pagetables:5496kB unstable:0kB bounce:0kB free_pcp:4148kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:28:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 04:28:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 04:28:14 oak-gw06 kernel: Node 0 DMA32: 6887*4kB (UEM) 6645*8kB (UEM) 1710*16kB (UEM) 5414*32kB (UEM) 2719*64kB (UEM) 209*128kB (UM) 5*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 483364kB Aug 15 04:28:14 oak-gw06 kernel: Node 0 Normal: 29613*4kB (UEM) 34369*8kB (UEM) 6911*16kB (UEM) 20291*32kB (UEM) 4382*64kB (UEM) 203*128kB (UM) 2*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1460236kB Aug 15 04:28:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 04:28:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 04:28:14 oak-gw06 kernel: 2106768 total pagecache pages Aug 15 04:28:14 oak-gw06 kernel: 16 pages in swap cache Aug 15 04:28:14 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 04:28:14 oak-gw06 kernel: Free swap = 4194036kB Aug 15 04:28:14 oak-gw06 kernel: Total swap = 4194300kB Aug 15 04:28:14 oak-gw06 kernel: 4194203 pages RAM Aug 15 04:28:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 04:28:14 oak-gw06 kernel: 127313 pages reserved Aug 15 04:28:14 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 04:28:14 oak-gw06 kernel: CPU: 6 PID: 6356 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 04:28:14 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 04:28:14 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 04:28:14 oak-gw06 kernel: 00000000000080d0 000000001ab771ac ffff8803d77eb808 ffffffff8168662f Aug 15 04:28:14 oak-gw06 kernel: ffff8803d77eb898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 04:28:14 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803d77eb868 000000001ab771ac Aug 15 04:28:14 oak-gw06 kernel: Call Trace: Aug 15 04:28:14 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 04:28:14 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 04:28:14 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 04:28:14 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 04:28:14 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 04:28:14 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 04:28:14 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 04:28:14 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 04:28:14 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 04:28:14 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 04:28:14 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 04:28:14 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 04:28:14 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 04:28:14 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 04:28:14 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 04:28:14 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 04:28:14 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 04:28:14 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 04:28:14 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 04:28:14 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 04:28:14 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 04:28:14 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 04:28:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:28:14 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 04:28:14 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:28:14 oak-gw06 kernel: Mem-Info: Aug 15 04:28:14 oak-gw06 kernel: active_anon:23441 inactive_anon:53142 isolated_anon:0#012 active_file:514653 inactive_file:2292624 isolated_file:0#012 unevictable:0 dirty:4589 writeback:2272 unstable:0#012 slab_reclaimable:32443 slab_unreclaimable:566913#012 mapped:10820 shmem:47126 pagetables:1693 bounce:0#012 free:488640 free_pcp:1297 free_cma:0 Aug 15 04:28:14 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 04:28:14 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 04:28:14 oak-gw06 kernel: Node 0 DMA32 free:485740kB min:69724kB low:87152kB high:104584kB active_anon:11408kB inactive_anon:35584kB active_file:392048kB inactive_file:1589580kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:3856kB writeback:216kB mapped:4872kB shmem:31268kB slab_reclaimable:18044kB slab_unreclaimable:304680kB kernel_stack:960kB pagetables:1276kB unstable:0kB bounce:0kB free_pcp:2640kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:28:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 04:28:14 oak-gw06 kernel: Node 0 Normal free:1472996kB min:323104kB low:403880kB high:484656kB active_anon:83136kB inactive_anon:176984kB active_file:1666564kB inactive_file:7559744kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:14112kB writeback:7028kB mapped:38408kB shmem:157236kB slab_reclaimable:111728kB slab_unreclaimable:1962684kB kernel_stack:4720kB pagetables:5496kB unstable:0kB bounce:0kB free_pcp:3716kB local_pcp:4kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:28:14 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 04:28:14 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 04:28:14 oak-gw06 kernel: Node 0 DMA32: 5934*4kB (UEM) 6620*8kB (UEM) 2253*16kB (UEM) 5415*32kB (UEM) 2719*64kB (UEM) 209*128kB (UM) 5*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 488072kB Aug 15 04:28:14 oak-gw06 kernel: Node 0 Normal: 29406*4kB (UE) 32792*8kB (UEM) 7849*16kB (UEM) 20292*32kB (UEM) 4382*64kB (UEM) 203*128kB (UM) 2*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1461832kB Aug 15 04:28:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 04:28:14 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 04:28:14 oak-gw06 kernel: 2106599 total pagecache pages Aug 15 04:28:14 oak-gw06 kernel: 16 pages in swap cache Aug 15 04:28:14 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 04:28:14 oak-gw06 kernel: Free swap = 4194036kB Aug 15 04:28:14 oak-gw06 kernel: Total swap = 4194300kB Aug 15 04:28:14 oak-gw06 kernel: 4194203 pages RAM Aug 15 04:28:14 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 04:28:14 oak-gw06 kernel: 127313 pages reserved Aug 15 04:33:15 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 04:33:15 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 04:33:15 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 04:33:15 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 04:33:15 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b858 ffffffff8168662f Aug 15 04:33:15 oak-gw06 kernel: ffff8800a4b8b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 04:33:15 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b8b8 00000000e48ba19f Aug 15 04:33:15 oak-gw06 kernel: Call Trace: Aug 15 04:33:15 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 04:33:15 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 04:33:15 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 04:33:15 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 04:33:15 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 04:33:15 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 04:33:15 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 04:33:15 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 04:33:15 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 04:33:15 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 04:33:15 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 04:33:15 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 04:33:15 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 04:33:15 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 04:33:15 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 04:33:15 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 04:33:15 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 04:33:15 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 04:33:15 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 04:33:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:33:15 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 04:33:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:33:15 oak-gw06 kernel: Mem-Info: Aug 15 04:33:15 oak-gw06 kernel: active_anon:25369 inactive_anon:53142 isolated_anon:0#012 active_file:632398 inactive_file:2448036 isolated_file:0#012 unevictable:0 dirty:16245 writeback:4885 unstable:0#012 slab_reclaimable:32359 slab_unreclaimable:562440#012 mapped:10831 shmem:47126 pagetables:1703 bounce:0#012 free:217965 free_pcp:518 free_cma:0 Aug 15 04:33:15 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 04:33:15 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 04:33:15 oak-gw06 kernel: Node 0 DMA32 free:234548kB min:69724kB low:87152kB high:104584kB active_anon:15600kB inactive_anon:35584kB active_file:469824kB inactive_file:1755840kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:9924kB writeback:3500kB mapped:4872kB shmem:31268kB slab_reclaimable:18024kB slab_unreclaimable:302748kB kernel_stack:976kB pagetables:1264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:33:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 04:33:15 oak-gw06 kernel: Node 0 Normal free:617520kB min:323104kB low:403880kB high:484656kB active_anon:86136kB inactive_anon:176984kB active_file:2059768kB inactive_file:8039424kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:54280kB writeback:16040kB mapped:38452kB shmem:157236kB slab_reclaimable:111412kB slab_unreclaimable:1946996kB kernel_stack:4720kB pagetables:5548kB unstable:0kB bounce:0kB free_pcp:2776kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:33:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 04:33:15 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 04:33:15 oak-gw06 kernel: Node 0 DMA32: 5330*4kB (UEM) 5549*8kB (UEM) 1173*16kB (UEM) 1348*32kB (UEM) 1359*64kB (UEM) 175*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 236992kB Aug 15 04:33:15 oak-gw06 kernel: Node 0 Normal: 22899*4kB (UEM) 23880*8kB (UEM) 5585*16kB (UEM) 3678*32kB (UEM) 1956*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 614876kB Aug 15 04:33:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 04:33:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 04:33:15 oak-gw06 kernel: 2118543 total pagecache pages Aug 15 04:33:15 oak-gw06 kernel: 16 pages in swap cache Aug 15 04:33:15 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 04:33:15 oak-gw06 kernel: Free swap = 4194036kB Aug 15 04:33:15 oak-gw06 kernel: Total swap = 4194300kB Aug 15 04:33:15 oak-gw06 kernel: 4194203 pages RAM Aug 15 04:33:15 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 04:33:15 oak-gw06 kernel: 127313 pages reserved Aug 15 04:33:15 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 04:33:15 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 04:33:15 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 04:33:15 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 04:33:15 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b808 ffffffff8168662f Aug 15 04:33:15 oak-gw06 kernel: ffff8800a4b8b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 04:33:15 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b868 00000000e48ba19f Aug 15 04:33:15 oak-gw06 kernel: Call Trace: Aug 15 04:33:15 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 04:33:15 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 04:33:15 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 04:33:15 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 04:33:15 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 04:33:15 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 04:33:15 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 04:33:15 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 04:33:15 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 04:33:15 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 04:33:15 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 04:33:15 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 04:33:15 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 04:33:15 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 04:33:15 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 04:33:15 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 04:33:15 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 04:33:15 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 04:33:15 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 04:33:15 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 04:33:15 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 04:33:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:33:15 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 04:33:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:33:15 oak-gw06 kernel: Mem-Info: Aug 15 04:33:15 oak-gw06 kernel: active_anon:25369 inactive_anon:53142 isolated_anon:0#012 active_file:631943 inactive_file:2448040 isolated_file:0#012 unevictable:0 dirty:15857 writeback:4885 unstable:0#012 slab_reclaimable:32359 slab_unreclaimable:562440#012 mapped:10831 shmem:47126 pagetables:1703 bounce:0#012 free:218548 free_pcp:655 free_cma:0 Aug 15 04:33:15 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 04:33:15 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 04:33:15 oak-gw06 kernel: Node 0 DMA32 free:239640kB min:69724kB low:87152kB high:104584kB active_anon:15600kB inactive_anon:35584kB active_file:469824kB inactive_file:1754832kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:9924kB writeback:3500kB mapped:4872kB shmem:31268kB slab_reclaimable:18024kB slab_unreclaimable:302748kB kernel_stack:976kB pagetables:1264kB unstable:0kB bounce:0kB free_pcp:1012kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:33:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 04:33:15 oak-gw06 kernel: Node 0 Normal free:614932kB min:323104kB low:403880kB high:484656kB active_anon:86136kB inactive_anon:176984kB active_file:2057948kB inactive_file:8039408kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:54280kB writeback:14100kB mapped:38452kB shmem:157236kB slab_reclaimable:111412kB slab_unreclaimable:1946996kB kernel_stack:4720kB pagetables:5548kB unstable:0kB bounce:0kB free_pcp:2496kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:33:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 04:33:15 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 04:33:15 oak-gw06 kernel: Node 0 DMA32: 5611*4kB (UEM) 5603*8kB (UEM) 1423*16kB (UEM) 1352*32kB (UEM) 1359*64kB (UEM) 175*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 242676kB Aug 15 04:33:15 oak-gw06 kernel: Node 0 Normal: 22645*4kB (UE) 23877*8kB (UEM) 5312*16kB (UE) 3670*32kB (UEM) 1951*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 608892kB Aug 15 04:33:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 04:33:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 04:33:15 oak-gw06 kernel: 2116227 total pagecache pages Aug 15 04:33:15 oak-gw06 kernel: 16 pages in swap cache Aug 15 04:33:15 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 04:33:15 oak-gw06 kernel: Free swap = 4194036kB Aug 15 04:33:15 oak-gw06 kernel: Total swap = 4194300kB Aug 15 04:33:15 oak-gw06 kernel: 4194203 pages RAM Aug 15 04:33:15 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 04:33:15 oak-gw06 kernel: 127313 pages reserved Aug 15 04:38:15 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 15 04:38:15 oak-gw06 kernel: CPU: 6 PID: 6370 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 04:38:15 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 04:38:15 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 04:38:15 oak-gw06 kernel: 00000000000080d0 000000007386397a ffff8803bc7ab858 ffffffff8168662f Aug 15 04:38:15 oak-gw06 kernel: ffff8803bc7ab8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 04:38:15 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803bc7ab8b8 000000007386397a Aug 15 04:38:15 oak-gw06 kernel: Call Trace: Aug 15 04:38:15 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 04:38:15 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 04:38:15 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 04:38:15 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 04:38:15 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 04:38:15 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 04:38:15 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 04:38:15 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 04:38:15 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 04:38:15 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 04:38:15 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 04:38:15 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 04:38:15 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 04:38:15 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 04:38:15 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 04:38:15 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 04:38:15 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 04:38:15 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 04:38:15 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 04:38:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:38:15 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 04:38:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:38:15 oak-gw06 kernel: Mem-Info: Aug 15 04:38:15 oak-gw06 kernel: active_anon:22367 inactive_anon:53142 isolated_anon:0#012 active_file:1297196 inactive_file:749701 isolated_file:0#012 unevictable:0 dirty:12348 writeback:1636 unstable:0#012 slab_reclaimable:32231 slab_unreclaimable:547103#012 mapped:10835 shmem:47126 pagetables:1699 bounce:0#012 free:1270532 free_pcp:623 free_cma:0 Aug 15 04:38:15 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 04:38:15 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 04:38:15 oak-gw06 kernel: Node 0 DMA32 free:1266964kB min:69724kB low:87152kB high:104584kB active_anon:13948kB inactive_anon:35584kB active_file:690480kB inactive_file:519884kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:7280kB writeback:476kB mapped:4884kB shmem:31268kB slab_reclaimable:17956kB slab_unreclaimable:291432kB kernel_stack:976kB pagetables:1268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:38:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 04:38:15 oak-gw06 kernel: Node 0 Normal free:3796000kB min:323104kB low:403880kB high:484656kB active_anon:75780kB inactive_anon:176984kB active_file:4498304kB inactive_file:2481260kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:41336kB writeback:2964kB mapped:38456kB shmem:157236kB slab_reclaimable:110968kB slab_unreclaimable:1896964kB kernel_stack:4704kB pagetables:5528kB unstable:0kB bounce:0kB free_pcp:2664kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:38:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 04:38:15 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 04:38:15 oak-gw06 kernel: Node 0 DMA32: 5633*4kB (UEM) 18234*8kB (UEM) 24722*16kB (UEM) 12304*32kB (UEM) 4163*64kB (UEM) 345*128kB (UM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1269300kB Aug 15 04:38:15 oak-gw06 kernel: Node 0 Normal: 28007*4kB (UEM) 71153*8kB (UEM) 103376*16kB (UEM) 33739*32kB (UEM) 5378*64kB (UEM) 250*128kB (UM) 3*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3791876kB Aug 15 04:38:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 04:38:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 04:38:15 oak-gw06 kernel: 2083970 total pagecache pages Aug 15 04:38:15 oak-gw06 kernel: 16 pages in swap cache Aug 15 04:38:15 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 04:38:15 oak-gw06 kernel: Free swap = 4194036kB Aug 15 04:38:15 oak-gw06 kernel: Total swap = 4194300kB Aug 15 04:38:15 oak-gw06 kernel: 4194203 pages RAM Aug 15 04:38:15 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 04:38:15 oak-gw06 kernel: 127313 pages reserved Aug 15 04:38:15 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 15 04:38:15 oak-gw06 kernel: CPU: 6 PID: 6370 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 04:38:15 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 04:38:15 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 04:38:15 oak-gw06 kernel: 00000000000080d0 000000007386397a ffff8803bc7ab808 ffffffff8168662f Aug 15 04:38:15 oak-gw06 kernel: ffff8803bc7ab898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 04:38:15 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8803bc7ab868 000000007386397a Aug 15 04:38:15 oak-gw06 kernel: Call Trace: Aug 15 04:38:15 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 04:38:15 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 04:38:15 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 04:38:15 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 04:38:15 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 04:38:15 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 04:38:15 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 04:38:15 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 04:38:15 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 04:38:15 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 04:38:15 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 04:38:15 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 04:38:15 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 04:38:15 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 04:38:15 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 04:38:15 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 04:38:15 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 04:38:15 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 04:38:15 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 04:38:15 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 04:38:15 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 04:38:15 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 04:38:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:38:15 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 04:38:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:38:15 oak-gw06 kernel: Mem-Info: Aug 15 04:38:15 oak-gw06 kernel: active_anon:22367 inactive_anon:53142 isolated_anon:0#012 active_file:1297196 inactive_file:754056 isolated_file:0#012 unevictable:0 dirty:12154 writeback:472 unstable:0#012 slab_reclaimable:32231 slab_unreclaimable:547103#012 mapped:10835 shmem:47126 pagetables:1699 bounce:0#012 free:1266020 free_pcp:609 free_cma:0 Aug 15 04:38:15 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 04:38:15 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 04:38:15 oak-gw06 kernel: Node 0 DMA32 free:1266964kB min:69724kB low:87152kB high:104584kB active_anon:13948kB inactive_anon:35584kB active_file:690480kB inactive_file:519884kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:7280kB writeback:476kB mapped:4884kB shmem:31268kB slab_reclaimable:17956kB slab_unreclaimable:291432kB kernel_stack:976kB pagetables:1268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:38:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 04:38:15 oak-gw06 kernel: Node 0 Normal free:3777748kB min:323104kB low:403880kB high:484656kB active_anon:75520kB inactive_anon:176984kB active_file:4498304kB inactive_file:2499460kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:42112kB writeback:1800kB mapped:38456kB shmem:157236kB slab_reclaimable:110968kB slab_unreclaimable:1896964kB kernel_stack:4704kB pagetables:5528kB unstable:0kB bounce:0kB free_pcp:2436kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:38:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 04:38:15 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 04:38:15 oak-gw06 kernel: Node 0 DMA32: 5633*4kB (UEM) 18234*8kB (UEM) 24722*16kB (UEM) 12304*32kB (UEM) 4163*64kB (UEM) 345*128kB (UM) 4*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1269300kB Aug 15 04:38:15 oak-gw06 kernel: Node 0 Normal: 27979*4kB (UEM) 69230*8kB (UEM) 103373*16kB (UEM) 33740*32kB (UEM) 5378*64kB (UEM) 250*128kB (UM) 3*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3776364kB Aug 15 04:38:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 04:38:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 04:38:15 oak-gw06 kernel: 2089208 total pagecache pages Aug 15 04:38:15 oak-gw06 kernel: 16 pages in swap cache Aug 15 04:38:15 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 04:38:15 oak-gw06 kernel: Free swap = 4194036kB Aug 15 04:38:15 oak-gw06 kernel: Total swap = 4194300kB Aug 15 04:38:15 oak-gw06 kernel: 4194203 pages RAM Aug 15 04:38:15 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 04:38:15 oak-gw06 kernel: 127313 pages reserved Aug 15 04:43:15 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 04:43:15 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 04:43:15 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 04:43:15 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 04:43:15 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b858 ffffffff8168662f Aug 15 04:43:15 oak-gw06 kernel: ffff8800a4b8b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 04:43:15 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b8b8 00000000e48ba19f Aug 15 04:43:15 oak-gw06 kernel: Call Trace: Aug 15 04:43:15 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 04:43:15 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 04:43:15 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 04:43:15 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 04:43:15 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 04:43:15 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 04:43:15 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 04:43:15 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 04:43:15 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 04:43:15 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 04:43:15 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 04:43:15 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 04:43:15 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 04:43:15 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 04:43:15 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 04:43:15 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 04:43:15 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 04:43:15 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 04:43:15 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 04:43:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:43:15 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 04:43:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:43:15 oak-gw06 kernel: Mem-Info: Aug 15 04:43:15 oak-gw06 kernel: active_anon:19646 inactive_anon:53142 isolated_anon:0#012 active_file:2050338 inactive_file:26786 isolated_file:0#012 unevictable:0 dirty:18142 writeback:3332 unstable:0#012 slab_reclaimable:32180 slab_unreclaimable:546468#012 mapped:10850 shmem:47126 pagetables:1674 bounce:0#012 free:1228805 free_pcp:389 free_cma:0 Aug 15 04:43:15 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 04:43:15 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 04:43:15 oak-gw06 kernel: Node 0 DMA32 free:1461496kB min:69724kB low:87152kB high:104584kB active_anon:11396kB inactive_anon:35584kB active_file:1012984kB inactive_file:5544kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:6544kB writeback:372kB mapped:4896kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:284920kB kernel_stack:976kB pagetables:1256kB unstable:0kB bounce:0kB free_pcp:636kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:43:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 04:43:15 oak-gw06 kernel: Node 0 Normal free:3438516kB min:323104kB low:403880kB high:484656kB active_anon:65140kB inactive_anon:176984kB active_file:7199500kB inactive_file:92028kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:66412kB writeback:11016kB mapped:38504kB shmem:157236kB slab_reclaimable:110780kB slab_unreclaimable:1900936kB kernel_stack:4720kB pagetables:5440kB unstable:0kB bounce:0kB free_pcp:1332kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:43:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 04:43:15 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 04:43:15 oak-gw06 kernel: Node 0 DMA32: 6554*4kB (UEM) 21167*8kB (UEM) 14507*16kB (UEM) 16417*32kB (UEM) 6230*64kB (UEM) 843*128kB (UM) 21*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1465008kB Aug 15 04:43:15 oak-gw06 kernel: Node 0 Normal: 23138*4kB (UE) 62232*8kB (UEM) 94435*16kB (UEM) 29031*32kB (UEM) 5665*64kB (UEM) 326*128kB (UM) 3*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3435416kB Aug 15 04:43:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 04:43:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 04:43:15 oak-gw06 kernel: 2112968 total pagecache pages Aug 15 04:43:15 oak-gw06 kernel: 16 pages in swap cache Aug 15 04:43:15 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 04:43:15 oak-gw06 kernel: Free swap = 4194036kB Aug 15 04:43:15 oak-gw06 kernel: Total swap = 4194300kB Aug 15 04:43:15 oak-gw06 kernel: 4194203 pages RAM Aug 15 04:43:15 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 04:43:15 oak-gw06 kernel: 127313 pages reserved Aug 15 04:43:15 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 04:43:15 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 04:43:15 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 04:43:15 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 04:43:15 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b808 ffffffff8168662f Aug 15 04:43:15 oak-gw06 kernel: ffff8800a4b8b898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 04:43:15 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b868 00000000e48ba19f Aug 15 04:43:15 oak-gw06 kernel: Call Trace: Aug 15 04:43:15 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 04:43:15 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 04:43:15 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 04:43:15 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 04:43:15 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 04:43:15 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 04:43:15 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 04:43:15 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 04:43:15 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 04:43:15 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 04:43:15 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 04:43:15 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 04:43:15 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 04:43:15 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 04:43:15 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 04:43:15 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 04:43:15 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 04:43:15 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 04:43:15 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 04:43:15 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 04:43:15 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 04:43:15 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 04:43:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:43:15 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 04:43:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:43:15 oak-gw06 kernel: Mem-Info: Aug 15 04:43:15 oak-gw06 kernel: active_anon:19264 inactive_anon:53142 isolated_anon:0#012 active_file:2060726 inactive_file:18023 isolated_file:0#012 unevictable:0 dirty:18239 writeback:3138 unstable:0#012 slab_reclaimable:32180 slab_unreclaimable:546536#012 mapped:10850 shmem:47126 pagetables:1674 bounce:0#012 free:1226077 free_pcp:428 free_cma:0 Aug 15 04:43:15 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 04:43:15 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 04:43:15 oak-gw06 kernel: Node 0 DMA32 free:1466080kB min:69724kB low:87152kB high:104584kB active_anon:11396kB inactive_anon:35584kB active_file:1012984kB inactive_file:5544kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:6544kB writeback:372kB mapped:4896kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:284920kB kernel_stack:976kB pagetables:1256kB unstable:0kB bounce:0kB free_pcp:888kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:43:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 04:43:15 oak-gw06 kernel: Node 0 Normal free:3411656kB min:323104kB low:403880kB high:484656kB active_anon:67220kB inactive_anon:176984kB active_file:7236160kB inactive_file:63688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:68352kB writeback:12956kB mapped:38504kB shmem:157236kB slab_reclaimable:110780kB slab_unreclaimable:1901208kB kernel_stack:4720kB pagetables:5440kB unstable:0kB bounce:0kB free_pcp:2644kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:43:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 04:43:15 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 04:43:15 oak-gw06 kernel: Node 0 DMA32: 7017*4kB (UEM) 21166*8kB (UEM) 14677*16kB (UEM) 16417*32kB (UEM) 6230*64kB (UEM) 843*128kB (UM) 21*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1469572kB Aug 15 04:43:15 oak-gw06 kernel: Node 0 Normal: 25817*4kB (UEM) 60971*8kB (UEM) 93819*16kB (UEM) 29043*32kB (UEM) 5665*64kB (UEM) 326*128kB (UM) 3*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3426572kB Aug 15 04:43:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 04:43:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 04:43:15 oak-gw06 kernel: 2108615 total pagecache pages Aug 15 04:43:15 oak-gw06 kernel: 16 pages in swap cache Aug 15 04:43:15 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 04:43:15 oak-gw06 kernel: Free swap = 4194036kB Aug 15 04:43:15 oak-gw06 kernel: Total swap = 4194300kB Aug 15 04:43:15 oak-gw06 kernel: 4194203 pages RAM Aug 15 04:43:15 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 04:43:15 oak-gw06 kernel: 127313 pages reserved Aug 15 04:48:16 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 04:48:16 oak-gw06 kernel: CPU: 6 PID: 6412 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 04:48:16 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 04:48:16 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 04:48:16 oak-gw06 kernel: 00000000000080d0 00000000a08abebe ffff880264e17858 ffffffff8168662f Aug 15 04:48:16 oak-gw06 kernel: ffff880264e178e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 15 04:48:16 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff880264e178e8 00000000a08abebe Aug 15 04:48:16 oak-gw06 kernel: Call Trace: Aug 15 04:48:16 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 04:48:16 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 04:48:16 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 04:48:16 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 04:48:16 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 04:48:16 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 04:48:16 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 04:48:16 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 04:48:16 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 04:48:16 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 04:48:16 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 04:48:16 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 04:48:16 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 04:48:16 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 04:48:16 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 04:48:16 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 04:48:16 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 04:48:16 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 04:48:16 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 04:48:16 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:48:16 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 04:48:16 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:48:16 oak-gw06 kernel: Mem-Info: Aug 15 04:48:16 oak-gw06 kernel: active_anon:24621 inactive_anon:53142 isolated_anon:0#012 active_file:2023754 inactive_file:27915 isolated_file:19#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:543059#012 mapped:10876 shmem:47126 pagetables:1699 bounce:0#012 free:1268832 free_pcp:61 free_cma:0 Aug 15 04:48:16 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 04:48:16 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 04:48:16 oak-gw06 kernel: Node 0 DMA32 free:1481888kB min:69724kB low:87152kB high:104584kB active_anon:10396kB inactive_anon:35584kB active_file:991364kB inactive_file:10652kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4888kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:284768kB kernel_stack:960kB pagetables:1072kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:48:16 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 04:48:16 oak-gw06 kernel: Node 0 Normal free:3576804kB min:323104kB low:403880kB high:484656kB active_anon:88608kB inactive_anon:176984kB active_file:7103652kB inactive_file:101008kB unevictable:0kB isolated(anon):0kB isolated(file):76kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:38616kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1887452kB kernel_stack:4736kB pagetables:5724kB unstable:0kB bounce:0kB free_pcp:440kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:48:16 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 04:48:16 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 04:48:16 oak-gw06 kernel: Node 0 DMA32: 5715*4kB (UEM) 7471*8kB (UEM) 21213*16kB (UEM) 16384*32kB (UEM) 6416*64kB (UEM) 923*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1482004kB Aug 15 04:48:16 oak-gw06 kernel: Node 0 Normal: 37447*4kB (UEM) 69870*8kB (UEM) 92019*16kB (UEM) 30844*32kB (UEM) 5720*64kB (UEM) 327*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3577020kB Aug 15 04:48:16 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 04:48:16 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 04:48:16 oak-gw06 kernel: 2098851 total pagecache pages Aug 15 04:48:16 oak-gw06 kernel: 16 pages in swap cache Aug 15 04:48:16 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 04:48:16 oak-gw06 kernel: Free swap = 4194036kB Aug 15 04:48:16 oak-gw06 kernel: Total swap = 4194300kB Aug 15 04:48:16 oak-gw06 kernel: 4194203 pages RAM Aug 15 04:48:16 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 04:48:16 oak-gw06 kernel: 127313 pages reserved Aug 15 04:48:16 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 04:48:16 oak-gw06 kernel: CPU: 6 PID: 6412 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 04:48:16 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 04:48:16 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 04:48:16 oak-gw06 kernel: 00000000000080d0 00000000a08abebe ffff880264e17808 ffffffff8168662f Aug 15 04:48:16 oak-gw06 kernel: ffff880264e17898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 04:48:16 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880264e17868 00000000a08abebe Aug 15 04:48:16 oak-gw06 kernel: Call Trace: Aug 15 04:48:16 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 04:48:16 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 04:48:16 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 04:48:16 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 04:48:16 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 04:48:16 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 04:48:16 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 04:48:16 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 04:48:16 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 04:48:16 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 04:48:16 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 04:48:16 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 04:48:16 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 04:48:16 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 04:48:16 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 04:48:16 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 04:48:16 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 04:48:16 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 04:48:16 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 04:48:16 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 04:48:16 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 04:48:16 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 04:48:16 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:48:16 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 04:48:16 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:48:16 oak-gw06 kernel: Mem-Info: Aug 15 04:48:16 oak-gw06 kernel: active_anon:24621 inactive_anon:53142 isolated_anon:0#012 active_file:2023689 inactive_file:27915 isolated_file:19#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:543059#012 mapped:10876 shmem:47126 pagetables:1699 bounce:0#012 free:1268843 free_pcp:159 free_cma:0 Aug 15 04:48:16 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 04:48:16 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 04:48:16 oak-gw06 kernel: Node 0 DMA32 free:1481984kB min:69724kB low:87152kB high:104584kB active_anon:10396kB inactive_anon:35584kB active_file:991364kB inactive_file:10652kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4888kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:284768kB kernel_stack:960kB pagetables:1072kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:48:16 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 04:48:16 oak-gw06 kernel: Node 0 Normal free:3577496kB min:323104kB low:403880kB high:484656kB active_anon:88088kB inactive_anon:176984kB active_file:7103392kB inactive_file:101008kB unevictable:0kB isolated(anon):0kB isolated(file):76kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:38616kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1887452kB kernel_stack:4736kB pagetables:5724kB unstable:0kB bounce:0kB free_pcp:348kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:48:16 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 04:48:16 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 04:48:16 oak-gw06 kernel: Node 0 DMA32: 5715*4kB (UEM) 7471*8kB (UEM) 21213*16kB (UEM) 16384*32kB (UEM) 6416*64kB (UEM) 923*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1482004kB Aug 15 04:48:16 oak-gw06 kernel: Node 0 Normal: 37571*4kB (UEM) 69888*8kB (UEM) 92019*16kB (UEM) 30844*32kB (UEM) 5720*64kB (UEM) 327*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3577660kB Aug 15 04:48:16 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 04:48:16 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 04:48:16 oak-gw06 kernel: 2098754 total pagecache pages Aug 15 04:48:16 oak-gw06 kernel: 16 pages in swap cache Aug 15 04:48:16 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 04:48:16 oak-gw06 kernel: Free swap = 4194036kB Aug 15 04:48:16 oak-gw06 kernel: Total swap = 4194300kB Aug 15 04:48:16 oak-gw06 kernel: 4194203 pages RAM Aug 15 04:48:16 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 04:48:16 oak-gw06 kernel: 127313 pages reserved Aug 15 04:53:15 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 04:53:15 oak-gw06 kernel: CPU: 6 PID: 6412 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 04:53:15 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 04:53:15 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 04:53:15 oak-gw06 kernel: 00000000000080d0 00000000a08abebe ffff880264e17858 ffffffff8168662f Aug 15 04:53:15 oak-gw06 kernel: ffff880264e178e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 04:53:15 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880264e178b8 00000000a08abebe Aug 15 04:53:15 oak-gw06 kernel: Call Trace: Aug 15 04:53:15 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 04:53:15 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 04:53:15 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 04:53:15 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 04:53:15 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 04:53:15 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 04:53:15 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 04:53:15 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 04:53:15 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 04:53:15 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 04:53:15 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 04:53:15 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 04:53:15 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 04:53:15 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 04:53:15 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 04:53:15 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 04:53:15 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 04:53:15 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 04:53:15 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 04:53:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:53:15 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 04:53:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:53:15 oak-gw06 kernel: Mem-Info: Aug 15 04:53:15 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2012777 inactive_file:11933 isolated_file:0#012 unevictable:0 dirty:0 writeback:1 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:539727#012 mapped:10621 shmem:47126 pagetables:1413 bounce:0#012 free:1311389 free_pcp:190 free_cma:0 Aug 15 04:53:15 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 04:53:15 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 04:53:15 oak-gw06 kernel: Node 0 DMA32 free:1500380kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:984592kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4764kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:283884kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:53:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 04:53:15 oak-gw06 kernel: Node 0 Normal free:3728540kB min:323104kB low:403880kB high:484656kB active_anon:40936kB inactive_anon:176984kB active_file:7066516kB inactive_file:47100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:4kB mapped:37720kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1875008kB kernel_stack:4720kB pagetables:4592kB unstable:0kB bounce:0kB free_pcp:796kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:53:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 04:53:15 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 04:53:15 oak-gw06 kernel: Node 0 DMA32: 7419*4kB (UEM) 8532*8kB (UEM) 21313*16kB (UEM) 16411*32kB (UEM) 6424*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1500412kB Aug 15 04:53:15 oak-gw06 kernel: Node 0 Normal: 61395*4kB (UEM) 75056*8kB (UEM) 92633*16kB (UEM) 30969*32kB (UEM) 5729*64kB (UEM) 329*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3728956kB Aug 15 04:53:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 04:53:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 04:53:15 oak-gw06 kernel: 2071856 total pagecache pages Aug 15 04:53:15 oak-gw06 kernel: 16 pages in swap cache Aug 15 04:53:15 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 04:53:15 oak-gw06 kernel: Free swap = 4194036kB Aug 15 04:53:15 oak-gw06 kernel: Total swap = 4194300kB Aug 15 04:53:15 oak-gw06 kernel: 4194203 pages RAM Aug 15 04:53:15 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 04:53:15 oak-gw06 kernel: 127313 pages reserved Aug 15 04:53:15 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 04:53:15 oak-gw06 kernel: CPU: 1 PID: 6412 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 04:53:15 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 04:53:15 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 04:53:15 oak-gw06 kernel: 00000000000080d0 00000000a08abebe ffff880264e17808 ffffffff8168662f Aug 15 04:53:15 oak-gw06 kernel: ffff880264e17898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 04:53:15 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880264e17868 00000000a08abebe Aug 15 04:53:15 oak-gw06 kernel: Call Trace: Aug 15 04:53:15 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 04:53:15 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 04:53:15 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 04:53:15 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 04:53:15 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 04:53:15 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 04:53:15 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 04:53:15 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 04:53:15 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 04:53:15 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 04:53:15 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 04:53:15 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 04:53:15 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 04:53:15 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 04:53:15 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 04:53:15 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 04:53:15 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 04:53:15 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 04:53:15 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 04:53:15 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 04:53:15 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 04:53:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:53:15 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 04:53:15 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:53:15 oak-gw06 kernel: Mem-Info: Aug 15 04:53:15 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2012712 inactive_file:11933 isolated_file:0#012 unevictable:0 dirty:0 writeback:1 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:539727#012 mapped:10621 shmem:47126 pagetables:1413 bounce:0#012 free:1311513 free_pcp:159 free_cma:0 Aug 15 04:53:15 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 04:53:15 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 04:53:15 oak-gw06 kernel: Node 0 DMA32 free:1500412kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:984592kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4764kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:283884kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:53:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 04:53:15 oak-gw06 kernel: Node 0 Normal free:3729376kB min:323104kB low:403880kB high:484656kB active_anon:40936kB inactive_anon:176984kB active_file:7066256kB inactive_file:47100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:4kB mapped:37720kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1875008kB kernel_stack:4720kB pagetables:4592kB unstable:0kB bounce:0kB free_pcp:688kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:53:15 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 04:53:15 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 04:53:15 oak-gw06 kernel: Node 0 DMA32: 7419*4kB (UEM) 8532*8kB (UEM) 21313*16kB (UEM) 16411*32kB (UEM) 6424*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1500412kB Aug 15 04:53:15 oak-gw06 kernel: Node 0 Normal: 61550*4kB (UEM) 75056*8kB (UEM) 92634*16kB (UEM) 30969*32kB (UEM) 5729*64kB (UEM) 329*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3729592kB Aug 15 04:53:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 04:53:15 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 04:53:15 oak-gw06 kernel: 2071759 total pagecache pages Aug 15 04:53:15 oak-gw06 kernel: 16 pages in swap cache Aug 15 04:53:15 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 04:53:15 oak-gw06 kernel: Free swap = 4194036kB Aug 15 04:53:15 oak-gw06 kernel: Total swap = 4194300kB Aug 15 04:53:15 oak-gw06 kernel: 4194203 pages RAM Aug 15 04:53:15 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 04:53:15 oak-gw06 kernel: 127313 pages reserved Aug 15 04:58:16 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 04:58:16 oak-gw06 kernel: CPU: 6 PID: 6412 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 04:58:16 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 04:58:16 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 04:58:16 oak-gw06 kernel: 00000000000080d0 00000000a08abebe ffff880264e17858 ffffffff8168662f Aug 15 04:58:16 oak-gw06 kernel: ffff880264e178e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 04:58:16 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880264e178b8 00000000a08abebe Aug 15 04:58:16 oak-gw06 kernel: Call Trace: Aug 15 04:58:16 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 04:58:16 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 04:58:16 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 04:58:16 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 04:58:16 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 04:58:16 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 04:58:16 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 04:58:16 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 04:58:16 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 04:58:16 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 04:58:16 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 04:58:16 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 04:58:16 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 04:58:16 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 04:58:16 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 04:58:16 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 04:58:16 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 04:58:16 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 04:58:16 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 04:58:16 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:58:16 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 04:58:16 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:58:16 oak-gw06 kernel: Mem-Info: Aug 15 04:58:16 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2012584 inactive_file:11942 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:538037#012 mapped:10631 shmem:47126 pagetables:1413 bounce:0#012 free:1313159 free_pcp:190 free_cma:0 Aug 15 04:58:16 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 04:58:16 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 04:58:16 oak-gw06 kernel: Node 0 DMA32 free:1500668kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:984592kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4764kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:283392kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:58:16 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 04:58:16 oak-gw06 kernel: Node 0 Normal free:3735356kB min:323104kB low:403880kB high:484656kB active_anon:41196kB inactive_anon:176984kB active_file:7065744kB inactive_file:47136kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:37760kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1868740kB kernel_stack:4720kB pagetables:4592kB unstable:0kB bounce:0kB free_pcp:852kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:58:16 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 04:58:16 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 04:58:16 oak-gw06 kernel: Node 0 DMA32: 7454*4kB (UEM) 8536*8kB (UEM) 21333*16kB (UEM) 16411*32kB (UEM) 6424*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1500904kB Aug 15 04:58:16 oak-gw06 kernel: Node 0 Normal: 61895*4kB (UEM) 75114*8kB (UEM) 92891*16kB (UEM) 30982*32kB (UEM) 5733*64kB (UEM) 329*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3736220kB Aug 15 04:58:16 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 04:58:16 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 04:58:16 oak-gw06 kernel: 2071638 total pagecache pages Aug 15 04:58:16 oak-gw06 kernel: 16 pages in swap cache Aug 15 04:58:16 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 04:58:16 oak-gw06 kernel: Free swap = 4194036kB Aug 15 04:58:16 oak-gw06 kernel: Total swap = 4194300kB Aug 15 04:58:16 oak-gw06 kernel: 4194203 pages RAM Aug 15 04:58:16 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 04:58:16 oak-gw06 kernel: 127313 pages reserved Aug 15 04:58:16 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 04:58:16 oak-gw06 kernel: CPU: 6 PID: 6412 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 04:58:16 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 04:58:16 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 04:58:16 oak-gw06 kernel: 00000000000080d0 00000000a08abebe ffff880264e17808 ffffffff8168662f Aug 15 04:58:16 oak-gw06 kernel: ffff880264e17898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 04:58:16 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880264e17868 00000000a08abebe Aug 15 04:58:16 oak-gw06 kernel: Call Trace: Aug 15 04:58:16 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 04:58:16 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 04:58:16 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 04:58:16 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 04:58:16 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 04:58:16 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 04:58:16 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 04:58:16 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 04:58:16 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 04:58:16 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 04:58:16 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 04:58:16 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 04:58:16 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 04:58:16 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 04:58:16 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 04:58:16 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 04:58:16 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 04:58:16 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 04:58:16 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 04:58:16 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 04:58:16 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 04:58:16 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 04:58:16 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:58:16 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 04:58:16 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 04:58:16 oak-gw06 kernel: Mem-Info: Aug 15 04:58:16 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2012584 inactive_file:11942 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:538037#012 mapped:10631 shmem:47126 pagetables:1413 bounce:0#012 free:1313304 free_pcp:31 free_cma:0 Aug 15 04:58:16 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 04:58:16 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 04:58:16 oak-gw06 kernel: Node 0 DMA32 free:1500668kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:984592kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4764kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:283392kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:58:16 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 04:58:16 oak-gw06 kernel: Node 0 Normal free:3736656kB min:323104kB low:403880kB high:484656kB active_anon:40676kB inactive_anon:176984kB active_file:7065744kB inactive_file:47136kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:37760kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1868740kB kernel_stack:4720kB pagetables:4592kB unstable:0kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 04:58:16 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 04:58:16 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 04:58:16 oak-gw06 kernel: Node 0 DMA32: 7454*4kB (UEM) 8536*8kB (UEM) 21333*16kB (UEM) 16411*32kB (UEM) 6424*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1500904kB Aug 15 04:58:16 oak-gw06 kernel: Node 0 Normal: 61992*4kB (UEM) 75114*8kB (UEM) 92891*16kB (UEM) 30982*32kB (UEM) 5733*64kB (UEM) 329*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3736608kB Aug 15 04:58:16 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 04:58:16 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 04:58:16 oak-gw06 kernel: 2071638 total pagecache pages Aug 15 04:58:16 oak-gw06 kernel: 16 pages in swap cache Aug 15 04:58:16 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 04:58:16 oak-gw06 kernel: Free swap = 4194036kB Aug 15 04:58:16 oak-gw06 kernel: Total swap = 4194300kB Aug 15 04:58:16 oak-gw06 kernel: 4194203 pages RAM Aug 15 04:58:16 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 04:58:16 oak-gw06 kernel: 127313 pages reserved Aug 15 05:03:17 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:03:17 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:03:17 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:03:17 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:03:17 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b858 ffffffff8168662f Aug 15 05:03:17 oak-gw06 kernel: ffff8800a4b8b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:03:17 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b8b8 00000000e48ba19f Aug 15 05:03:17 oak-gw06 kernel: Call Trace: Aug 15 05:03:17 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:03:17 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:03:17 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:03:17 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:03:17 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 05:03:17 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 05:03:17 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:03:17 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:03:17 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:03:17 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:03:17 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:03:17 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:03:17 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:03:17 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:03:17 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:03:17 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:03:17 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:03:17 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:03:17 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:03:17 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:03:17 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:03:17 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:03:17 oak-gw06 kernel: Mem-Info: Aug 15 05:03:17 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2012461 inactive_file:12183 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:533733#012 mapped:10643 shmem:47126 pagetables:1414 bounce:0#012 free:1317719 free_pcp:31 free_cma:0 Aug 15 05:03:17 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:03:17 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:03:17 oak-gw06 kernel: Node 0 DMA32 free:1501060kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:984592kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4764kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:283244kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:03:17 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:03:17 oak-gw06 kernel: Node 0 Normal free:3753180kB min:323104kB low:403880kB high:484656kB active_anon:41196kB inactive_anon:176984kB active_file:7065252kB inactive_file:48100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:37808kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1851672kB kernel_stack:4720kB pagetables:4596kB unstable:0kB bounce:0kB free_pcp:312kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:03:17 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:03:17 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:03:17 oak-gw06 kernel: Node 0 DMA32: 7463*4kB (UEM) 8538*8kB (UEM) 21338*16kB (UEM) 16412*32kB (UEM) 6424*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1501068kB Aug 15 05:03:17 oak-gw06 kernel: Node 0 Normal: 62601*4kB (UEM) 75264*8kB (UEM) 93594*16kB (UEM) 31005*32kB (UEM) 5744*64kB (UEM) 332*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3753316kB Aug 15 05:03:17 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:03:17 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:03:17 oak-gw06 kernel: 2071754 total pagecache pages Aug 15 05:03:17 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:03:17 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:03:17 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:03:17 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:03:17 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:03:17 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:03:17 oak-gw06 kernel: 127313 pages reserved Aug 15 05:03:17 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:03:17 oak-gw06 kernel: CPU: 7 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:03:17 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:03:17 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:03:17 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b808 ffffffff8168662f Aug 15 05:03:17 oak-gw06 kernel: ffff8800a4b8b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:03:17 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b868 00000000e48ba19f Aug 15 05:03:17 oak-gw06 kernel: Call Trace: Aug 15 05:03:17 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:03:17 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:03:17 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:03:17 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:03:17 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 05:03:17 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 05:03:17 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 05:03:17 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 05:03:17 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:03:17 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:03:17 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:03:17 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:03:17 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:03:17 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:03:17 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:03:17 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:03:17 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:03:17 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:03:17 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:03:17 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:03:17 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:03:17 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:03:17 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:03:17 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:03:17 oak-gw06 kernel: Mem-Info: Aug 15 05:03:17 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2012333 inactive_file:12183 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:533733#012 mapped:10643 shmem:47126 pagetables:1414 bounce:0#012 free:1317764 free_pcp:31 free_cma:0 Aug 15 05:03:17 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:03:17 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:03:17 oak-gw06 kernel: Node 0 DMA32 free:1501068kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:984592kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4764kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:283244kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:03:17 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:03:17 oak-gw06 kernel: Node 0 Normal free:3753724kB min:323104kB low:403880kB high:484656kB active_anon:40936kB inactive_anon:176984kB active_file:7064740kB inactive_file:48100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:37808kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1851672kB kernel_stack:4720kB pagetables:4596kB unstable:0kB bounce:0kB free_pcp:140kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:03:17 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:03:17 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:03:17 oak-gw06 kernel: Node 0 DMA32: 7463*4kB (UEM) 8538*8kB (UEM) 21338*16kB (UEM) 16412*32kB (UEM) 6424*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1501068kB Aug 15 05:03:17 oak-gw06 kernel: Node 0 Normal: 62750*4kB (UEM) 75269*8kB (UEM) 93594*16kB (UEM) 31005*32kB (UEM) 5744*64kB (UEM) 332*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3753952kB Aug 15 05:03:17 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:03:17 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:03:17 oak-gw06 kernel: 2071626 total pagecache pages Aug 15 05:03:17 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:03:17 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:03:17 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:03:17 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:03:17 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:03:17 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:03:17 oak-gw06 kernel: 127313 pages reserved Aug 15 05:08:18 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:08:18 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:08:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:08:18 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:08:18 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b858 ffffffff8168662f Aug 15 05:08:18 oak-gw06 kernel: ffff8800a4b8b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:08:18 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b8b8 00000000e48ba19f Aug 15 05:08:18 oak-gw06 kernel: Call Trace: Aug 15 05:08:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:08:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:08:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:08:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:08:18 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 05:08:18 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 05:08:18 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:08:18 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:08:18 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:08:18 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:08:18 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:08:18 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:08:18 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:08:18 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:08:18 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:08:18 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:08:18 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:08:18 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:08:18 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:08:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:08:18 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:08:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:08:18 oak-gw06 kernel: Mem-Info: Aug 15 05:08:18 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2012205 inactive_file:12186 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:532352#012 mapped:10657 shmem:47126 pagetables:1414 bounce:0#012 free:1319091 free_pcp:62 free_cma:0 Aug 15 05:08:18 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:08:18 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:08:18 oak-gw06 kernel: Node 0 DMA32 free:1501580kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:984592kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4776kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:282592kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:08:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:08:18 oak-gw06 kernel: Node 0 Normal free:3758576kB min:323104kB low:403880kB high:484656kB active_anon:40936kB inactive_anon:176984kB active_file:7064228kB inactive_file:48112kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:37852kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1846800kB kernel_stack:4720kB pagetables:4596kB unstable:0kB bounce:0kB free_pcp:380kB local_pcp:4kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:08:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:08:18 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:08:18 oak-gw06 kernel: Node 0 DMA32: 7482*4kB (UEM) 8540*8kB (UEM) 21371*16kB (UEM) 16411*32kB (UEM) 6425*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1501720kB Aug 15 05:08:18 oak-gw06 kernel: Node 0 Normal: 63053*4kB (UEM) 75323*8kB (UEM) 93789*16kB (UEM) 31010*32kB (UEM) 5747*64kB (UEM) 332*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3759068kB Aug 15 05:08:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:08:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:08:18 oak-gw06 kernel: 2071503 total pagecache pages Aug 15 05:08:18 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:08:18 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:08:18 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:08:18 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:08:18 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:08:18 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:08:18 oak-gw06 kernel: 127313 pages reserved Aug 15 05:08:18 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:08:18 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:08:18 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:08:18 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:08:18 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b808 ffffffff8168662f Aug 15 05:08:18 oak-gw06 kernel: ffff8800a4b8b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:08:18 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b868 00000000e48ba19f Aug 15 05:08:18 oak-gw06 kernel: Call Trace: Aug 15 05:08:18 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:08:18 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:08:18 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:08:18 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:08:18 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 05:08:18 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 05:08:18 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 05:08:18 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 05:08:18 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:08:18 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:08:18 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:08:18 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:08:18 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:08:18 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:08:18 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:08:18 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:08:18 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:08:18 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:08:18 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:08:18 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:08:18 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:08:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:08:18 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:08:18 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:08:18 oak-gw06 kernel: Mem-Info: Aug 15 05:08:18 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2012075 inactive_file:12186 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:532352#012 mapped:10657 shmem:47126 pagetables:1414 bounce:0#012 free:1319244 free_pcp:31 free_cma:0 Aug 15 05:08:18 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:08:18 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:08:18 oak-gw06 kernel: Node 0 DMA32 free:1501580kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:984592kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4776kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:282592kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:08:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:08:18 oak-gw06 kernel: Node 0 Normal free:3759504kB min:323104kB low:403880kB high:484656kB active_anon:40676kB inactive_anon:176984kB active_file:7063708kB inactive_file:48112kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:37852kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1846800kB kernel_stack:4720kB pagetables:4596kB unstable:0kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:08:18 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:08:18 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:08:18 oak-gw06 kernel: Node 0 DMA32: 7482*4kB (UEM) 8540*8kB (UEM) 21371*16kB (UEM) 16411*32kB (UEM) 6425*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1501720kB Aug 15 05:08:18 oak-gw06 kernel: Node 0 Normal: 63084*4kB (UEM) 75387*8kB (UEM) 93789*16kB (UEM) 31010*32kB (UEM) 5747*64kB (UEM) 332*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3759704kB Aug 15 05:08:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:08:18 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:08:18 oak-gw06 kernel: 2071406 total pagecache pages Aug 15 05:08:18 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:08:18 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:08:18 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:08:18 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:08:18 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:08:18 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:08:18 oak-gw06 kernel: 127313 pages reserved Aug 15 05:13:19 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:13:19 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:13:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:13:19 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:13:19 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b858 ffffffff8168662f Aug 15 05:13:19 oak-gw06 kernel: ffff8800a4b8b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:13:19 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b8b8 00000000e48ba19f Aug 15 05:13:19 oak-gw06 kernel: Call Trace: Aug 15 05:13:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:13:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:13:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:13:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:13:19 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 05:13:19 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 05:13:19 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:13:19 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:13:19 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:13:19 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:13:19 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:13:19 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:13:19 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:13:19 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:13:19 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:13:19 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:13:19 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:13:19 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:13:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:13:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:13:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:13:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:13:19 oak-gw06 kernel: Mem-Info: Aug 15 05:13:19 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2012015 inactive_file:12193 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:532071#012 mapped:10670 shmem:47126 pagetables:1414 bounce:0#012 free:1319467 free_pcp:128 free_cma:0 Aug 15 05:13:19 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:13:19 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:13:19 oak-gw06 kernel: Node 0 DMA32 free:1501720kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:984592kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4780kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:282572kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:480kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:13:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:13:19 oak-gw06 kernel: Node 0 Normal free:3759572kB min:323104kB low:403880kB high:484656kB active_anon:41196kB inactive_anon:176984kB active_file:7063468kB inactive_file:48140kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:37900kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1845696kB kernel_stack:4736kB pagetables:4596kB unstable:0kB bounce:0kB free_pcp:220kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:13:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:13:19 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:13:19 oak-gw06 kernel: Node 0 DMA32: 7483*4kB (UEM) 8540*8kB (UEM) 21372*16kB (UEM) 16411*32kB (UEM) 6425*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1501740kB Aug 15 05:13:19 oak-gw06 kernel: Node 0 Normal: 63092*4kB (UEM) 75409*8kB (UEM) 93846*16kB (UEM) 31013*32kB (UEM) 5747*64kB (UEM) 332*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3760920kB Aug 15 05:13:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:13:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:13:19 oak-gw06 kernel: 2071351 total pagecache pages Aug 15 05:13:19 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:13:19 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:13:19 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:13:19 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:13:19 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:13:19 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:13:19 oak-gw06 kernel: 127313 pages reserved Aug 15 05:13:19 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:13:19 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:13:19 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:13:19 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:13:19 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b808 ffffffff8168662f Aug 15 05:13:19 oak-gw06 kernel: ffff8800a4b8b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:13:19 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b868 00000000e48ba19f Aug 15 05:13:19 oak-gw06 kernel: Call Trace: Aug 15 05:13:19 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:13:19 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:13:19 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:13:19 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:13:19 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 05:13:19 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 05:13:19 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 05:13:19 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 05:13:19 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:13:19 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:13:19 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:13:19 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:13:19 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:13:19 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:13:19 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:13:19 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:13:19 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:13:19 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:13:19 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:13:19 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:13:19 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:13:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:13:19 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:13:19 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:13:19 oak-gw06 kernel: Mem-Info: Aug 15 05:13:19 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2011950 inactive_file:12193 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:532071#012 mapped:10670 shmem:47126 pagetables:1414 bounce:0#012 free:1319686 free_pcp:62 free_cma:0 Aug 15 05:13:19 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:13:19 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:13:19 oak-gw06 kernel: Node 0 DMA32 free:1501720kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:984592kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4780kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:282572kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:13:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:13:19 oak-gw06 kernel: Node 0 Normal free:3760492kB min:323104kB low:403880kB high:484656kB active_anon:41456kB inactive_anon:176984kB active_file:7063208kB inactive_file:48140kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:37900kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1845696kB kernel_stack:4736kB pagetables:4596kB unstable:0kB bounce:0kB free_pcp:420kB local_pcp:4kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:13:19 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:13:19 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:13:19 oak-gw06 kernel: Node 0 DMA32: 7487*4kB (UEM) 8598*8kB (UEM) 21372*16kB (UEM) 16411*32kB (UEM) 6425*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1502220kB Aug 15 05:13:19 oak-gw06 kernel: Node 0 Normal: 63048*4kB (UEM) 75468*8kB (UEM) 93847*16kB (UEM) 31012*32kB (UEM) 5748*64kB (UEM) 332*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3761264kB Aug 15 05:13:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:13:19 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:13:19 oak-gw06 kernel: 2071254 total pagecache pages Aug 15 05:13:19 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:13:19 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:13:19 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:13:19 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:13:19 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:13:19 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:13:19 oak-gw06 kernel: 127313 pages reserved Aug 15 05:18:20 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:18:20 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:18:20 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:18:20 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:18:20 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b858 ffffffff8168662f Aug 15 05:18:20 oak-gw06 kernel: ffff8800a4b8b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:18:20 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b8b8 00000000e48ba19f Aug 15 05:18:20 oak-gw06 kernel: Call Trace: Aug 15 05:18:20 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:18:20 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:18:20 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:18:20 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:18:20 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 05:18:20 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 05:18:20 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:18:20 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:18:20 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:18:20 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:18:20 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:18:20 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:18:20 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:18:20 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:18:20 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:18:20 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:18:20 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:18:20 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:18:20 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:18:20 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:18:20 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:18:20 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:18:20 oak-gw06 kernel: Mem-Info: Aug 15 05:18:20 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2011694 inactive_file:12198 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:531826#012 mapped:10680 shmem:47126 pagetables:1414 bounce:0#012 free:1320225 free_pcp:31 free_cma:0 Aug 15 05:18:20 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:18:20 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:18:20 oak-gw06 kernel: Node 0 DMA32 free:1502300kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:984112kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4780kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:282492kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:18:20 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:18:20 oak-gw06 kernel: Node 0 Normal free:3762076kB min:323104kB low:403880kB high:484656kB active_anon:41196kB inactive_anon:176984kB active_file:7062664kB inactive_file:48160kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:37940kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1844796kB kernel_stack:4720kB pagetables:4596kB unstable:0kB bounce:0kB free_pcp:272kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:18:20 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:18:20 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:18:20 oak-gw06 kernel: Node 0 DMA32: 7487*4kB (UEM) 8598*8kB (UEM) 21375*16kB (UEM) 16410*32kB (UEM) 6426*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1502300kB Aug 15 05:18:20 oak-gw06 kernel: Node 0 Normal: 63113*4kB (UEM) 75515*8kB (UEM) 93884*16kB (UEM) 31016*32kB (UEM) 5749*64kB (UEM) 332*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3762684kB Aug 15 05:18:20 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:18:20 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:18:20 oak-gw06 kernel: 2071003 total pagecache pages Aug 15 05:18:20 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:18:20 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:18:20 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:18:20 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:18:20 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:18:20 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:18:20 oak-gw06 kernel: 127313 pages reserved Aug 15 05:18:20 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:18:20 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:18:20 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:18:20 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:18:20 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b808 ffffffff8168662f Aug 15 05:18:20 oak-gw06 kernel: ffff8800a4b8b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:18:20 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b868 00000000e48ba19f Aug 15 05:18:20 oak-gw06 kernel: Call Trace: Aug 15 05:18:20 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:18:20 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:18:20 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:18:20 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:18:20 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 05:18:20 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 05:18:20 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 05:18:20 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 05:18:20 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:18:20 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:18:20 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:18:20 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:18:20 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:18:20 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:18:20 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:18:20 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:18:20 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:18:20 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:18:20 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:18:20 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:18:20 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:18:20 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:18:20 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:18:20 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:18:20 oak-gw06 kernel: Mem-Info: Aug 15 05:18:20 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2011568 inactive_file:12198 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:531826#012 mapped:10680 shmem:47126 pagetables:1414 bounce:0#012 free:1320323 free_pcp:31 free_cma:0 Aug 15 05:18:20 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:18:20 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:18:20 oak-gw06 kernel: Node 0 DMA32 free:1502804kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:983608kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4780kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:282492kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:18:20 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:18:20 oak-gw06 kernel: Node 0 Normal free:3761952kB min:323104kB low:403880kB high:484656kB active_anon:40936kB inactive_anon:176984kB active_file:7062664kB inactive_file:48160kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:37940kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1844796kB kernel_stack:4720kB pagetables:4596kB unstable:0kB bounce:0kB free_pcp:252kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:18:20 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:18:20 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:18:20 oak-gw06 kernel: Node 0 DMA32: 7611*4kB (UEM) 8600*8kB (UEM) 21375*16kB (UEM) 16410*32kB (UEM) 6426*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1502812kB Aug 15 05:18:20 oak-gw06 kernel: Node 0 Normal: 63082*4kB (UEM) 75515*8kB (UEM) 93884*16kB (UEM) 31017*32kB (UEM) 5749*64kB (UEM) 332*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3762592kB Aug 15 05:18:20 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:18:20 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:18:20 oak-gw06 kernel: 2070815 total pagecache pages Aug 15 05:18:20 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:18:20 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:18:20 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:18:20 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:18:20 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:18:20 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:18:20 oak-gw06 kernel: 127313 pages reserved Aug 15 05:23:21 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 05:23:21 oak-gw06 kernel: CPU: 6 PID: 6412 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:23:21 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:23:21 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:23:21 oak-gw06 kernel: 00000000000080d0 00000000a08abebe ffff880264e17858 ffffffff8168662f Aug 15 05:23:21 oak-gw06 kernel: ffff880264e178e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:23:21 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880264e178b8 00000000a08abebe Aug 15 05:23:21 oak-gw06 kernel: Call Trace: Aug 15 05:23:21 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:23:21 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:23:21 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:23:21 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:23:21 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 05:23:21 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 05:23:21 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:23:21 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:23:21 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:23:21 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:23:21 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:23:21 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:23:21 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:23:21 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:23:21 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:23:21 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:23:21 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:23:21 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:23:21 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:23:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:23:21 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:23:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:23:21 oak-gw06 kernel: Mem-Info: Aug 15 05:23:21 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2011440 inactive_file:12204 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:531515#012 mapped:10691 shmem:47126 pagetables:1414 bounce:0#012 free:1320542 free_pcp:128 free_cma:0 Aug 15 05:23:21 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:23:21 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:23:21 oak-gw06 kernel: Node 0 DMA32 free:1502844kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:983600kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4780kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:282428kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:23:21 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:23:21 oak-gw06 kernel: Node 0 Normal free:3762732kB min:323104kB low:403880kB high:484656kB active_anon:40676kB inactive_anon:176984kB active_file:7062160kB inactive_file:48184kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:37984kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1843616kB kernel_stack:4720kB pagetables:4596kB unstable:0kB bounce:0kB free_pcp:1184kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:23:21 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:23:21 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:23:21 oak-gw06 kernel: Node 0 DMA32: 7615*4kB (UEM) 8600*8kB (UEM) 21376*16kB (UEM) 16411*32kB (UEM) 6426*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1502876kB Aug 15 05:23:21 oak-gw06 kernel: Node 0 Normal: 63139*4kB (UEM) 75517*8kB (UEM) 93940*16kB (UEM) 31022*32kB (UEM) 5749*64kB (UEM) 332*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3763892kB Aug 15 05:23:21 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:23:21 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:23:21 oak-gw06 kernel: 2070757 total pagecache pages Aug 15 05:23:21 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:23:21 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:23:21 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:23:21 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:23:21 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:23:21 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:23:21 oak-gw06 kernel: 127313 pages reserved Aug 15 05:23:21 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 15 05:23:21 oak-gw06 kernel: CPU: 6 PID: 6412 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:23:21 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:23:21 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:23:21 oak-gw06 kernel: 00000000000080d0 00000000a08abebe ffff880264e17808 ffffffff8168662f Aug 15 05:23:21 oak-gw06 kernel: ffff880264e17898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 05:23:21 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880264e17868 00000000a08abebe Aug 15 05:23:21 oak-gw06 kernel: Call Trace: Aug 15 05:23:21 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:23:21 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:23:21 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 05:23:21 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:23:21 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:23:21 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 05:23:21 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 05:23:21 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 05:23:21 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 05:23:21 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:23:21 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:23:21 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:23:21 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:23:21 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:23:21 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:23:21 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:23:21 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:23:21 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:23:21 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:23:21 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:23:21 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:23:21 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:23:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:23:21 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:23:21 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:23:21 oak-gw06 kernel: Mem-Info: Aug 15 05:23:21 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2011310 inactive_file:12204 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:531515#012 mapped:10691 shmem:47126 pagetables:1414 bounce:0#012 free:1320692 free_pcp:128 free_cma:0 Aug 15 05:23:21 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:23:21 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:23:21 oak-gw06 kernel: Node 0 DMA32 free:1502844kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:983600kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4780kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:282428kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:23:21 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:23:21 oak-gw06 kernel: Node 0 Normal free:3764032kB min:323104kB low:403880kB high:484656kB active_anon:40676kB inactive_anon:176984kB active_file:7061640kB inactive_file:48184kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:37984kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1843616kB kernel_stack:4720kB pagetables:4596kB unstable:0kB bounce:0kB free_pcp:752kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:23:21 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:23:21 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:23:21 oak-gw06 kernel: Node 0 DMA32: 7615*4kB (UEM) 8600*8kB (UEM) 21376*16kB (UEM) 16411*32kB (UEM) 6426*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1502876kB Aug 15 05:23:21 oak-gw06 kernel: Node 0 Normal: 63158*4kB (UEM) 75576*8kB (UEM) 93945*16kB (UEM) 31022*32kB (UEM) 5749*64kB (UEM) 332*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3764520kB Aug 15 05:23:21 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:23:21 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:23:21 oak-gw06 kernel: 2070660 total pagecache pages Aug 15 05:23:21 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:23:21 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:23:21 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:23:21 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:23:21 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:23:21 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:23:21 oak-gw06 kernel: 127313 pages reserved Aug 15 05:28:22 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:28:22 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:28:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:28:22 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:28:22 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b858 ffffffff8168662f Aug 15 05:28:22 oak-gw06 kernel: ffff8800a4b8b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:28:22 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b8b8 00000000e48ba19f Aug 15 05:28:22 oak-gw06 kernel: Call Trace: Aug 15 05:28:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:28:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:28:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:28:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:28:22 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 05:28:22 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 05:28:22 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:28:22 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:28:22 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:28:22 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:28:22 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:28:22 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:28:22 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:28:22 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:28:22 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:28:22 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:28:22 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:28:22 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:28:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:28:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:28:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:28:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:28:22 oak-gw06 kernel: Mem-Info: Aug 15 05:28:22 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2011184 inactive_file:12210 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:531136#012 mapped:10699 shmem:47126 pagetables:1414 bounce:0#012 free:1321372 free_pcp:31 free_cma:0 Aug 15 05:28:22 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:28:22 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:28:22 oak-gw06 kernel: Node 0 DMA32 free:1502972kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:983600kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4780kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:282356kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:28:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:28:22 oak-gw06 kernel: Node 0 Normal free:3766296kB min:323104kB low:403880kB high:484656kB active_anon:41196kB inactive_anon:176984kB active_file:7061136kB inactive_file:48208kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:38016kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1842172kB kernel_stack:4720kB pagetables:4596kB unstable:0kB bounce:0kB free_pcp:280kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:28:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:28:22 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:28:22 oak-gw06 kernel: Node 0 DMA32: 7615*4kB (UEM) 8601*8kB (UEM) 21382*16kB (UEM) 16411*32kB (UEM) 6426*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1502980kB Aug 15 05:28:22 oak-gw06 kernel: Node 0 Normal: 63405*4kB (UEM) 75589*8kB (UEM) 94002*16kB (UEM) 31025*32kB (UEM) 5749*64kB (UEM) 333*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3766748kB Aug 15 05:28:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:28:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:28:22 oak-gw06 kernel: 2070506 total pagecache pages Aug 15 05:28:22 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:28:22 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:28:22 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:28:22 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:28:22 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:28:22 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:28:22 oak-gw06 kernel: 127313 pages reserved Aug 15 05:28:22 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:28:22 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:28:22 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:28:22 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:28:22 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b808 ffffffff8168662f Aug 15 05:28:22 oak-gw06 kernel: ffff8800a4b8b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:28:22 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b868 00000000e48ba19f Aug 15 05:28:22 oak-gw06 kernel: Call Trace: Aug 15 05:28:22 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:28:22 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:28:22 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:28:22 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:28:22 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 05:28:22 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 05:28:22 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 05:28:22 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 05:28:22 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:28:22 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:28:22 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:28:22 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:28:22 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:28:22 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:28:22 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:28:22 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:28:22 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:28:22 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:28:22 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:28:22 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:28:22 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:28:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:28:22 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:28:22 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:28:22 oak-gw06 kernel: Mem-Info: Aug 15 05:28:22 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2011054 inactive_file:12210 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:531136#012 mapped:10699 shmem:47126 pagetables:1414 bounce:0#012 free:1321522 free_pcp:31 free_cma:0 Aug 15 05:28:22 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:28:22 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:28:22 oak-gw06 kernel: Node 0 DMA32 free:1502972kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:983600kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4780kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:282356kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:28:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:28:22 oak-gw06 kernel: Node 0 Normal free:3767224kB min:323104kB low:403880kB high:484656kB active_anon:40676kB inactive_anon:176984kB active_file:7060616kB inactive_file:48208kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:38016kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1842172kB kernel_stack:4720kB pagetables:4596kB unstable:0kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:28:22 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:28:22 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:28:22 oak-gw06 kernel: Node 0 DMA32: 7615*4kB (UEM) 8601*8kB (UEM) 21382*16kB (UEM) 16411*32kB (UEM) 6426*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1502980kB Aug 15 05:28:22 oak-gw06 kernel: Node 0 Normal: 63419*4kB (UEM) 75622*8kB (UEM) 94002*16kB (UEM) 31029*32kB (UEM) 5750*64kB (UEM) 333*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3767260kB Aug 15 05:28:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:28:22 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:28:22 oak-gw06 kernel: 2070409 total pagecache pages Aug 15 05:28:22 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:28:22 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:28:22 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:28:22 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:28:22 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:28:22 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:28:22 oak-gw06 kernel: 127313 pages reserved Aug 15 05:33:23 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:33:23 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:33:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:33:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:33:23 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b858 ffffffff8168662f Aug 15 05:33:23 oak-gw06 kernel: ffff8800a4b8b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:33:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b8b8 00000000e48ba19f Aug 15 05:33:23 oak-gw06 kernel: Call Trace: Aug 15 05:33:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:33:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:33:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:33:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:33:23 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 05:33:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 05:33:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:33:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:33:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:33:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:33:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:33:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:33:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:33:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:33:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:33:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:33:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:33:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:33:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:33:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:33:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:33:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:33:23 oak-gw06 kernel: Mem-Info: Aug 15 05:33:23 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2010868 inactive_file:12221 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:530631#012 mapped:10716 shmem:47126 pagetables:1414 bounce:0#012 free:1322123 free_pcp:31 free_cma:0 Aug 15 05:33:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:33:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:33:23 oak-gw06 kernel: Node 0 DMA32 free:1503548kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:983096kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4792kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:282276kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:33:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:33:23 oak-gw06 kernel: Node 0 Normal free:3768384kB min:323104kB low:403880kB high:484656kB active_anon:40676kB inactive_anon:176984kB active_file:7060376kB inactive_file:48252kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:38072kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1840232kB kernel_stack:4720kB pagetables:4596kB unstable:0kB bounce:0kB free_pcp:744kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:33:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:33:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:33:23 oak-gw06 kernel: Node 0 DMA32: 7752*4kB (UEM) 8604*8kB (UEM) 21386*16kB (UEM) 16411*32kB (UEM) 6426*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1503616kB Aug 15 05:33:23 oak-gw06 kernel: Node 0 Normal: 63458*4kB (UEM) 75625*8kB (UEM) 94093*16kB (UEM) 31033*32kB (UEM) 5752*64kB (UEM) 333*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3769152kB Aug 15 05:33:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:33:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:33:23 oak-gw06 kernel: 2070169 total pagecache pages Aug 15 05:33:23 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:33:23 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:33:23 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:33:23 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:33:23 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:33:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:33:23 oak-gw06 kernel: 127313 pages reserved Aug 15 05:33:23 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:33:23 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:33:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:33:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:33:23 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b808 ffffffff8168662f Aug 15 05:33:23 oak-gw06 kernel: ffff8800a4b8b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:33:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b868 00000000e48ba19f Aug 15 05:33:23 oak-gw06 kernel: Call Trace: Aug 15 05:33:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:33:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:33:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:33:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:33:23 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 05:33:23 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 05:33:23 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 05:33:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 05:33:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:33:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:33:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:33:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:33:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:33:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:33:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:33:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:33:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:33:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:33:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:33:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:33:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:33:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:33:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:33:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:33:23 oak-gw06 kernel: Mem-Info: Aug 15 05:33:23 oak-gw06 kernel: active_anon:12618 inactive_anon:53142 isolated_anon:0#012 active_file:2010803 inactive_file:12221 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:32176 slab_unreclaimable:530631#012 mapped:10716 shmem:47126 pagetables:1414 bounce:0#012 free:1322281 free_pcp:31 free_cma:0 Aug 15 05:33:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:33:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:33:23 oak-gw06 kernel: Node 0 DMA32 free:1503548kB min:69724kB low:87152kB high:104584kB active_anon:9796kB inactive_anon:35584kB active_file:983096kB inactive_file:632kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:4792kB shmem:31268kB slab_reclaimable:17940kB slab_unreclaimable:282276kB kernel_stack:960kB pagetables:1060kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:33:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:33:23 oak-gw06 kernel: Node 0 Normal free:3769684kB min:323104kB low:403880kB high:484656kB active_anon:40676kB inactive_anon:176984kB active_file:7060116kB inactive_file:48252kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:38072kB shmem:157236kB slab_reclaimable:110764kB slab_unreclaimable:1840232kB kernel_stack:4720kB pagetables:4596kB unstable:0kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:33:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:33:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:33:23 oak-gw06 kernel: Node 0 DMA32: 7752*4kB (UEM) 8604*8kB (UEM) 21386*16kB (UEM) 16411*32kB (UEM) 6426*64kB (UEM) 924*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1503616kB Aug 15 05:33:23 oak-gw06 kernel: Node 0 Normal: 63574*4kB (UEM) 75625*8kB (UEM) 94092*16kB (UEM) 31035*32kB (UEM) 5752*64kB (UEM) 333*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3769664kB Aug 15 05:33:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:33:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:33:23 oak-gw06 kernel: 2070072 total pagecache pages Aug 15 05:33:23 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:33:23 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:33:23 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:33:23 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:33:23 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:33:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:33:23 oak-gw06 kernel: 127313 pages reserved Aug 15 05:38:23 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:38:23 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:38:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:38:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:38:23 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b858 ffffffff8168662f Aug 15 05:38:23 oak-gw06 kernel: ffff8800a4b8b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:38:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b8b8 00000000e48ba19f Aug 15 05:38:23 oak-gw06 kernel: Call Trace: Aug 15 05:38:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:38:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:38:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:38:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:38:23 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 05:38:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 05:38:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:38:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:38:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:38:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:38:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:38:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:38:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:38:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:38:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:38:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:38:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:38:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:38:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:38:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:38:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:38:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:38:23 oak-gw06 kernel: Mem-Info: Aug 15 05:38:23 oak-gw06 kernel: active_anon:25551 inactive_anon:53142 isolated_anon:0#012 active_file:2049533 inactive_file:17273 isolated_file:0#012 unevictable:0 dirty:29216 writeback:9878 unstable:0#012 slab_reclaimable:32156 slab_unreclaimable:532317#012 mapped:10985 shmem:47126 pagetables:1702 bounce:0#012 free:1260274 free_pcp:311 free_cma:0 Aug 15 05:38:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:38:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:38:23 oak-gw06 kernel: Node 0 DMA32 free:1359776kB min:69724kB low:87152kB high:104584kB active_anon:10580kB inactive_anon:35584kB active_file:1120064kB inactive_file:7584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:18344kB writeback:1520kB mapped:4916kB shmem:31268kB slab_reclaimable:17908kB slab_unreclaimable:281696kB kernel_stack:960kB pagetables:1068kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:38:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:38:23 oak-gw06 kernel: Node 0 Normal free:3664356kB min:323104kB low:403880kB high:484656kB active_anon:91624kB inactive_anon:176984kB active_file:7081968kB inactive_file:57348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:98520kB writeback:33724kB mapped:39024kB shmem:157236kB slab_reclaimable:110716kB slab_unreclaimable:1847556kB kernel_stack:4736kB pagetables:5740kB unstable:0kB bounce:0kB free_pcp:1116kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:38:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:38:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:38:23 oak-gw06 kernel: Node 0 DMA32: 6091*4kB (UEM) 6543*8kB (UEM) 7770*16kB (UEM) 16464*32kB (UEM) 7181*64kB (UEM) 1269*128kB (UM) 49*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1362436kB Aug 15 05:38:23 oak-gw06 kernel: Node 0 Normal: 29271*4kB (UE) 90100*8kB (UEM) 87441*16kB (UEM) 30306*32kB (UEM) 6265*64kB (UEM) 412*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3661452kB Aug 15 05:38:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:38:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:38:23 oak-gw06 kernel: 2114232 total pagecache pages Aug 15 05:38:23 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:38:23 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:38:23 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:38:23 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:38:23 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:38:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:38:23 oak-gw06 kernel: 127313 pages reserved Aug 15 05:38:23 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:38:23 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:38:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:38:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:38:23 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b808 ffffffff8168662f Aug 15 05:38:23 oak-gw06 kernel: ffff8800a4b8b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:38:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b868 00000000e48ba19f Aug 15 05:38:23 oak-gw06 kernel: Call Trace: Aug 15 05:38:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:38:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:38:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:38:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:38:23 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 05:38:23 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 05:38:23 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 05:38:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 05:38:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:38:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:38:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:38:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:38:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:38:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:38:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:38:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:38:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:38:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:38:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:38:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:38:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:38:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:38:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:38:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:38:23 oak-gw06 kernel: Mem-Info: Aug 15 05:38:23 oak-gw06 kernel: active_anon:25551 inactive_anon:53142 isolated_anon:0#012 active_file:2054112 inactive_file:15944 isolated_file:0#012 unevictable:0 dirty:21310 writeback:9751 unstable:0#012 slab_reclaimable:32156 slab_unreclaimable:532317#012 mapped:10985 shmem:47126 pagetables:1702 bounce:0#012 free:1258807 free_pcp:580 free_cma:0 Aug 15 05:38:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:38:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:38:23 oak-gw06 kernel: Node 0 DMA32 free:1364376kB min:69724kB low:87152kB high:104584kB active_anon:10580kB inactive_anon:35584kB active_file:1123592kB inactive_file:3048kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:12328kB writeback:4528kB mapped:4916kB shmem:31268kB slab_reclaimable:17908kB slab_unreclaimable:281696kB kernel_stack:960kB pagetables:1068kB unstable:0kB bounce:0kB free_pcp:1536kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:38:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:38:23 oak-gw06 kernel: Node 0 Normal free:3665120kB min:323104kB low:403880kB high:484656kB active_anon:91624kB inactive_anon:176984kB active_file:7076508kB inactive_file:65408kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:72136kB writeback:15100kB mapped:39024kB shmem:157236kB slab_reclaimable:110716kB slab_unreclaimable:1847284kB kernel_stack:4736kB pagetables:5740kB unstable:0kB bounce:0kB free_pcp:3628kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:38:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:38:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:38:23 oak-gw06 kernel: Node 0 DMA32: 6284*4kB (UEM) 6647*8kB (UEM) 8009*16kB (UEM) 16465*32kB (UEM) 7182*64kB (UEM) 1269*128kB (UM) 49*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1367960kB Aug 15 05:38:23 oak-gw06 kernel: Node 0 Normal: 34164*4kB (UEM) 88737*8kB (UEM) 87578*16kB (UEM) 30315*32kB (UEM) 6265*64kB (UEM) 412*128kB (UM) 4*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3672600kB Aug 15 05:38:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:38:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:38:23 oak-gw06 kernel: 2110594 total pagecache pages Aug 15 05:38:23 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:38:23 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:38:23 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:38:23 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:38:23 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:38:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:38:23 oak-gw06 kernel: 127313 pages reserved Aug 15 05:43:23 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:43:23 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:43:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:43:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:43:23 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b858 ffffffff8168662f Aug 15 05:43:23 oak-gw06 kernel: ffff8800a4b8b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:43:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b8b8 00000000e48ba19f Aug 15 05:43:23 oak-gw06 kernel: Call Trace: Aug 15 05:43:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:43:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:43:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:43:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:43:23 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 05:43:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 05:43:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:43:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:43:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:43:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:43:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:43:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:43:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:43:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:43:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:43:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:43:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:43:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:43:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:43:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:43:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:43:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:43:23 oak-gw06 kernel: Mem-Info: Aug 15 05:43:23 oak-gw06 kernel: active_anon:24624 inactive_anon:53142 isolated_anon:0#012 active_file:2050322 inactive_file:18394 isolated_file:0#012 unevictable:0 dirty:19890 writeback:1915 unstable:0#012 slab_reclaimable:32148 slab_unreclaimable:532196#012 mapped:10995 shmem:47126 pagetables:1715 bounce:0#012 free:1259878 free_pcp:321 free_cma:0 Aug 15 05:43:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:43:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:43:23 oak-gw06 kernel: Node 0 DMA32 free:1304464kB min:69724kB low:87152kB high:104584kB active_anon:13632kB inactive_anon:35584kB active_file:1176440kB inactive_file:6836kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:13956kB writeback:192kB mapped:4916kB shmem:31268kB slab_reclaimable:17908kB slab_unreclaimable:284228kB kernel_stack:960kB pagetables:1064kB unstable:0kB bounce:0kB free_pcp:228kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:43:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:43:23 oak-gw06 kernel: Node 0 Normal free:3716372kB min:323104kB low:403880kB high:484656kB active_anon:85384kB inactive_anon:176984kB active_file:7033332kB inactive_file:62400kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:65992kB writeback:9408kB mapped:39064kB shmem:157236kB slab_reclaimable:110684kB slab_unreclaimable:1844540kB kernel_stack:4752kB pagetables:5796kB unstable:0kB bounce:0kB free_pcp:1216kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:43:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:43:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:43:23 oak-gw06 kernel: Node 0 DMA32: 7843*4kB (UEM) 7143*8kB (UEM) 1832*16kB (UEM) 14582*32kB (UEM) 7673*64kB (UEM) 1594*128kB (UM) 98*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1305156kB Aug 15 05:43:23 oak-gw06 kernel: Node 0 Normal: 40172*4kB (UEM) 91793*8kB (UEM) 84129*16kB (UEM) 29986*32kB (UEM) 6884*64kB (UEM) 557*128kB (UM) 9*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3714824kB Aug 15 05:43:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:43:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:43:23 oak-gw06 kernel: 2107367 total pagecache pages Aug 15 05:43:23 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:43:23 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:43:23 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:43:23 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:43:23 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:43:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:43:23 oak-gw06 kernel: 127313 pages reserved Aug 15 05:43:23 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:43:23 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:43:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:43:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:43:23 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b808 ffffffff8168662f Aug 15 05:43:23 oak-gw06 kernel: ffff8800a4b8b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:43:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b868 00000000e48ba19f Aug 15 05:43:23 oak-gw06 kernel: Call Trace: Aug 15 05:43:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:43:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:43:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:43:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:43:23 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 05:43:23 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 05:43:23 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 05:43:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 05:43:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:43:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:43:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:43:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:43:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:43:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:43:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:43:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:43:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:43:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:43:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:43:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:43:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:43:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:43:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:43:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:43:23 oak-gw06 kernel: Mem-Info: Aug 15 05:43:23 oak-gw06 kernel: active_anon:24624 inactive_anon:53142 isolated_anon:0#012 active_file:2056973 inactive_file:14079 isolated_file:0#012 unevictable:0 dirty:21054 writeback:2400 unstable:0#012 slab_reclaimable:32148 slab_unreclaimable:532196#012 mapped:10995 shmem:47126 pagetables:1715 bounce:0#012 free:1258595 free_pcp:285 free_cma:0 Aug 15 05:43:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:43:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:43:23 oak-gw06 kernel: Node 0 DMA32 free:1304464kB min:69724kB low:87152kB high:104584kB active_anon:13632kB inactive_anon:35584kB active_file:1178960kB inactive_file:4316kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:13956kB writeback:192kB mapped:4916kB shmem:31268kB slab_reclaimable:17908kB slab_unreclaimable:284228kB kernel_stack:960kB pagetables:1064kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:43:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:43:23 oak-gw06 kernel: Node 0 Normal free:3711676kB min:323104kB low:403880kB high:484656kB active_anon:85384kB inactive_anon:176984kB active_file:7049192kB inactive_file:53820kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:71812kB writeback:9408kB mapped:39064kB shmem:157236kB slab_reclaimable:110684kB slab_unreclaimable:1844540kB kernel_stack:4752kB pagetables:5796kB unstable:0kB bounce:0kB free_pcp:1480kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:43:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:43:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:43:23 oak-gw06 kernel: Node 0 DMA32: 7900*4kB (UEM) 7143*8kB (UEM) 1832*16kB (UEM) 14582*32kB (UEM) 7673*64kB (UEM) 1594*128kB (UM) 98*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1305384kB Aug 15 05:43:23 oak-gw06 kernel: Node 0 Normal: 38500*4kB (UEM) 91793*8kB (UEM) 84228*16kB (UEM) 29986*32kB (UEM) 6884*64kB (UEM) 557*128kB (UM) 9*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3709720kB Aug 15 05:43:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:43:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:43:23 oak-gw06 kernel: 2109501 total pagecache pages Aug 15 05:43:23 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:43:23 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:43:23 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:43:23 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:43:23 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:43:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:43:23 oak-gw06 kernel: 127313 pages reserved Aug 15 05:48:23 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:48:23 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:48:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:48:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:48:23 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b858 ffffffff8168662f Aug 15 05:48:23 oak-gw06 kernel: ffff8800a4b8b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 15 05:48:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b8b8 00000000e48ba19f Aug 15 05:48:23 oak-gw06 kernel: Call Trace: Aug 15 05:48:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:48:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:48:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:48:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:48:23 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 15 05:48:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 15 05:48:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:48:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:48:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:48:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:48:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:48:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:48:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:48:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:48:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:48:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:48:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:48:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:48:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:48:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:48:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:48:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:48:23 oak-gw06 kernel: Mem-Info: Aug 15 05:48:23 oak-gw06 kernel: active_anon:24627 inactive_anon:53142 isolated_anon:0#012 active_file:1979531 inactive_file:42089 isolated_file:0#012 unevictable:0 dirty:14100 writeback:5531 unstable:0#012 slab_reclaimable:32140 slab_unreclaimable:530196#012 mapped:11322 shmem:47126 pagetables:1694 bounce:0#012 free:1307675 free_pcp:1192 free_cma:0 Aug 15 05:48:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:48:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:48:23 oak-gw06 kernel: Node 0 DMA32 free:1358668kB min:69724kB low:87152kB high:104584kB active_anon:15112kB inactive_anon:35584kB active_file:1091172kB inactive_file:23756kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:9384kB writeback:1088kB mapped:5020kB shmem:31268kB slab_reclaimable:17908kB slab_unreclaimable:281504kB kernel_stack:960kB pagetables:1776kB unstable:0kB bounce:0kB free_pcp:2724kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:48:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:48:23 oak-gw06 kernel: Node 0 Normal free:3841824kB min:323104kB low:403880kB high:484656kB active_anon:83396kB inactive_anon:176984kB active_file:6826952kB inactive_file:151620kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:46240kB writeback:17932kB mapped:40268kB shmem:157236kB slab_reclaimable:110652kB slab_unreclaimable:1839264kB kernel_stack:4736kB pagetables:5000kB unstable:0kB bounce:0kB free_pcp:2768kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:48:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:48:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:48:23 oak-gw06 kernel: Node 0 DMA32: 4725*4kB (UEM) 6594*8kB (UEM) 4465*16kB (UEM) 14089*32kB (UEM) 7853*64kB (UEM) 1787*128kB (UM) 143*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1362388kB Aug 15 05:48:23 oak-gw06 kernel: Node 0 Normal: 48654*4kB (UEM) 94413*8kB (UEM) 81623*16kB (UEM) 30869*32kB (UEM) 7705*64kB (UEM) 719*128kB (UM) 16*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3832944kB Aug 15 05:48:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:48:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:48:23 oak-gw06 kernel: 2066881 total pagecache pages Aug 15 05:48:23 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:48:23 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:48:23 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:48:23 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:48:23 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:48:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:48:23 oak-gw06 kernel: 127313 pages reserved Aug 15 05:48:23 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 15 05:48:23 oak-gw06 kernel: CPU: 6 PID: 6384 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 15 05:48:23 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 15 05:48:23 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 15 05:48:23 oak-gw06 kernel: 00000000000080d0 00000000e48ba19f ffff8800a4b8b808 ffffffff8168662f Aug 15 05:48:23 oak-gw06 kernel: ffff8800a4b8b898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 15 05:48:23 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800a4b8b868 00000000e48ba19f Aug 15 05:48:23 oak-gw06 kernel: Call Trace: Aug 15 05:48:23 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 15 05:48:23 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 15 05:48:23 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 15 05:48:23 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 15 05:48:23 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 15 05:48:23 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 15 05:48:23 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 15 05:48:23 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 15 05:48:23 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 15 05:48:23 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 15 05:48:23 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 15 05:48:23 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 15 05:48:23 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 15 05:48:23 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 15 05:48:23 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 15 05:48:23 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 15 05:48:23 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 15 05:48:23 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 15 05:48:23 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 15 05:48:23 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 15 05:48:23 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 15 05:48:23 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 15 05:48:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:48:23 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 15 05:48:23 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 15 05:48:23 oak-gw06 kernel: Mem-Info: Aug 15 05:48:23 oak-gw06 kernel: active_anon:24627 inactive_anon:53142 isolated_anon:0#012 active_file:1981470 inactive_file:46174 isolated_file:0#012 unevictable:0 dirty:15082 writeback:5619 unstable:0#012 slab_reclaimable:32140 slab_unreclaimable:530196#012 mapped:11322 shmem:47126 pagetables:1694 bounce:0#012 free:1300273 free_pcp:444 free_cma:0 Aug 15 05:48:23 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 15 05:48:23 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 15 05:48:23 oak-gw06 kernel: Node 0 DMA32 free:1362356kB min:69724kB low:87152kB high:104584kB active_anon:15112kB inactive_anon:35584kB active_file:1095288kB inactive_file:22156kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:10596kB writeback:2604kB mapped:5020kB shmem:31268kB slab_reclaimable:17908kB slab_unreclaimable:281504kB kernel_stack:960kB pagetables:1776kB unstable:0kB bounce:0kB free_pcp:224kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:48:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 15 05:48:23 oak-gw06 kernel: Node 0 Normal free:3815912kB min:323104kB low:403880kB high:484656kB active_anon:83396kB inactive_anon:176984kB active_file:6838132kB inactive_file:156820kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:49732kB writeback:22200kB mapped:40268kB shmem:157236kB slab_reclaimable:110652kB slab_unreclaimable:1839264kB kernel_stack:4736kB pagetables:5000kB unstable:0kB bounce:0kB free_pcp:1992kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 15 05:48:23 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 05:48:23 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 15 05:48:23 oak-gw06 kernel: Node 0 DMA32: 5101*4kB (UEM) 6324*8kB (UEM) 4572*16kB (UEM) 14091*32kB (UEM) 7853*64kB (UEM) 1787*128kB (UM) 143*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1363508kB Aug 15 05:48:23 oak-gw06 kernel: Node 0 Normal: 44329*4kB (UEM) 94416*8kB (UEM) 81350*16kB (UEM) 30869*32kB (UEM) 7705*64kB (UEM) 719*128kB (UM) 16*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3811300kB Aug 15 05:48:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 15 05:48:23 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 15 05:48:23 oak-gw06 kernel: 2071103 total pagecache pages Aug 15 05:48:23 oak-gw06 kernel: 16 pages in swap cache Aug 15 05:48:23 oak-gw06 kernel: Swap cache stats: add 66, delete 50, find 0/0 Aug 15 05:48:23 oak-gw06 kernel: Free swap = 4194036kB Aug 15 05:48:23 oak-gw06 kernel: Total swap = 4194300kB Aug 15 05:48:23 oak-gw06 kernel: 4194203 pages RAM Aug 15 05:48:23 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 15 05:48:23 oak-gw06 kernel: 127313 pages reserved Aug 16 12:34:12 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502911451/real 1502911451] req@ffff8802f7e98000 x1566271273297872/t0(0) o4->oak-OST0027-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/448 e 23 to 1 dl 1502912052 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 16 12:34:12 oak-gw06 kernel: Lustre: oak-OST0017-osc-ffff88041b99c000: Connection to oak-OST0017 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 16 12:34:12 oak-gw06 kernel: Lustre: Skipped 9 previous similar messages Aug 16 12:34:12 oak-gw06 kernel: Lustre: oak-OST0017-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 16 12:34:12 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 24 previous similar messages Aug 16 12:34:14 oak-gw06 kernel: Lustre: oak-OST0009-osc-ffff88041b99c000: Connection to oak-OST0009 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 16 12:34:14 oak-gw06 kernel: Lustre: Skipped 5 previous similar messages Aug 16 12:34:17 oak-gw06 kernel: Lustre: oak-OST000b-osc-ffff88041b99c000: Connection to oak-OST000b (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 16 12:34:17 oak-gw06 kernel: Lustre: oak-OST000b-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 16 12:34:17 oak-gw06 kernel: Lustre: Skipped 6 previous similar messages Aug 16 12:34:27 oak-gw06 kernel: Lustre: oak-OST0003-osc-ffff88041b99c000: Connection to oak-OST0003 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 16 12:34:27 oak-gw06 kernel: Lustre: oak-OST002b-osc-ffff88041b99c000: Connection to oak-OST002b (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 16 12:34:27 oak-gw06 kernel: Lustre: Skipped 4 previous similar messages Aug 16 12:34:27 oak-gw06 kernel: Lustre: oak-OST002b-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 16 12:34:27 oak-gw06 kernel: Lustre: Skipped 4 previous similar messages Aug 16 12:44:13 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502912052/real 1502912052] req@ffff88023cb7cf00 x1566271273301424/t0(0) o4->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/448 e 24 to 1 dl 1502912653 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Aug 16 12:44:13 oak-gw06 kernel: Lustre: oak-OST0027-osc-ffff88041b99c000: Connection to oak-OST0027 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 16 12:44:13 oak-gw06 kernel: Lustre: Skipped 11 previous similar messages Aug 16 12:44:13 oak-gw06 kernel: Lustre: oak-OST0017-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 16 12:44:13 oak-gw06 kernel: Lustre: Skipped 11 previous similar messages Aug 16 12:44:13 oak-gw06 kernel: Lustre: 1764:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 46 previous similar messages Aug 16 12:44:33 oak-gw06 kernel: Lustre: oak-OST0023-osc-ffff88041b99c000: Connection to oak-OST0023 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 16 12:44:33 oak-gw06 kernel: Lustre: Skipped 21 previous similar messages Aug 16 12:44:33 oak-gw06 kernel: Lustre: oak-OST0023-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 16 12:44:33 oak-gw06 kernel: Lustre: Skipped 22 previous similar messages Aug 16 12:47:33 oak-gw06 kernel: Lustre: 1767:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502912097/real 1502912097] req@ffff880062d9d800 x1566271273344608/t0(0) o400->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1502912853 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 16 12:47:33 oak-gw06 kernel: Lustre: 1767:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 151 previous similar messages Aug 16 12:53:23 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502912447/real 1502912447] req@ffff8803b43cc000 x1566271273356960/t0(0) o400->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1502913203 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 16 12:53:23 oak-gw06 kernel: Lustre: 1763:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Aug 16 12:54:14 oak-gw06 kernel: Lustre: oak-OST0017-osc-ffff88041b99c000: Connection to oak-OST0017 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 16 12:54:14 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 16 12:54:14 oak-gw06 kernel: Lustre: oak-OST0027-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 16 13:03:59 oak-gw06 kernel: Lustre: 1767:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502913832/real 1502913832] req@ffff88028da96d00 x1566271273460256/t0(0) o103->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:17/18 lens 328/224 e 0 to 1 dl 1502913839 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Aug 16 13:03:59 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1502913832/real 0] req@ffff880420b99800 x1566271273460192/t0(0) o103->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:17/18 lens 328/224 e 0 to 1 dl 1502913839 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 16 13:03:59 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1502913832/real 0] req@ffff88028da94600 x1566271273460208/t0(0) o103->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:17/18 lens 328/224 e 0 to 1 dl 1502913839 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 16 13:03:59 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1502913832/real 0] req@ffff88028da95200 x1566271273460496/t0(0) o103->oak-OST0003-osc-ffff88041b99c000@10.0.2.102@o2ib5:17/18 lens 328/224 e 0 to 1 dl 1502913839 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 16 13:03:59 oak-gw06 kernel: Lustre: 1769:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1502913832/real 0] req@ffff8802dd40c000 x1566271273460544/t0(0) o103->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:17/18 lens 328/224 e 0 to 1 dl 1502913839 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 16 13:03:59 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 190 previous similar messages Aug 16 13:03:59 oak-gw06 kernel: Lustre: 1762:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 190 previous similar messages Aug 16 13:03:59 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 190 previous similar messages Aug 16 13:03:59 oak-gw06 kernel: Lustre: 1769:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 190 previous similar messages Aug 16 13:03:59 oak-gw06 kernel: Lustre: oak-OST001f-osc-ffff88041b99c000: Connection to oak-OST001f (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 16 13:03:59 oak-gw06 kernel: Lustre: Skipped 23 previous similar messages Aug 16 13:03:59 oak-gw06 kernel: Lustre: 1767:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Aug 16 13:04:54 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3126:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 12 seconds Aug 16 13:04:54 oak-gw06 kernel: LNetError: 1754:0:(o2iblnd_cb.c:3189:kiblnd_check_conns()) Timed out RDMA with 10.0.2.102@o2ib5 (62): c: 0, oc: 0, rc: 8 Aug 16 13:08:37 oak-gw06 kernel: Lustre: oak-OST0029-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 16 13:08:37 oak-gw06 kernel: Lustre: Skipped 23 previous similar messages Aug 16 13:08:58 oak-gw06 kernel: Lustre: oak-OST001b-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 16 13:08:58 oak-gw06 kernel: Lustre: Skipped 3 previous similar messages Aug 16 13:13:53 oak-gw06 kernel: LustreError: 11-0: oak-OST0009-osc-ffff88041b99c000: operation ost_write to node 10.0.2.101@o2ib5 failed: rc = -107 Aug 16 13:13:53 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 16 13:13:55 oak-gw06 kernel: LustreError: 11-0: oak-OST0007-osc-ffff88041b99c000: operation ldlm_enqueue to node 10.0.2.101@o2ib5 failed: rc = -107 Aug 16 13:13:55 oak-gw06 kernel: LustreError: 1761:0:(import.c:671:ptlrpc_connect_import()) already connecting Aug 16 13:13:57 oak-gw06 kernel: LustreError: 11-0: oak-OST000f-osc-ffff88041b99c000: operation ldlm_enqueue to node 10.0.2.101@o2ib5 failed: rc = -107 Aug 16 13:13:57 oak-gw06 kernel: LustreError: Skipped 11 previous similar messages Aug 16 13:13:57 oak-gw06 kernel: LustreError: 1761:0:(import.c:671:ptlrpc_connect_import()) already connecting Aug 16 13:13:57 oak-gw06 kernel: LustreError: 1761:0:(import.c:671:ptlrpc_connect_import()) Skipped 10 previous similar messages Aug 16 13:13:59 oak-gw06 kernel: LustreError: 1761:0:(import.c:671:ptlrpc_connect_import()) already connecting Aug 16 13:13:59 oak-gw06 kernel: LustreError: 1761:0:(import.c:671:ptlrpc_connect_import()) Skipped 3 previous similar messages Aug 16 13:13:59 oak-gw06 kernel: LustreError: 11-0: oak-OST0005-osc-ffff88041b99c000: operation ost_write to node 10.0.2.101@o2ib5 failed: rc = -107 Aug 16 13:13:59 oak-gw06 kernel: LustreError: Skipped 9 previous similar messages Aug 16 13:14:01 oak-gw06 kernel: LustreError: 1761:0:(import.c:671:ptlrpc_connect_import()) already connecting Aug 16 13:14:01 oak-gw06 kernel: LustreError: 1761:0:(import.c:671:ptlrpc_connect_import()) Skipped 3 previous similar messages Aug 16 13:14:01 oak-gw06 kernel: LustreError: 167-0: oak-OST0009-osc-ffff88041b99c000: This client was evicted by oak-OST0009; in progress operations using this service will fail. Aug 16 13:14:01 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Aug 16 13:14:01 oak-gw06 kernel: LustreError: 13111:0:(ldlm_resource.c:882:ldlm_resource_complain()) oak-OST0009-osc-ffff88041b99c000: namespace resource [0x3d7bcf:0x0:0x0].0x0 (ffff88019732a9c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 16 13:14:01 oak-gw06 kernel: LustreError: 13111:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x3d7bcf:0x0:0x0].0x0 (ffff88019732a9c0) refcount = 2 Aug 16 13:14:01 oak-gw06 kernel: LustreError: 13111:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 16 13:14:01 oak-gw06 kernel: LustreError: 13111:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: oak-OST0009-osc-ffff88041b99c000 lock: ffff88041f548800/0xf077f1a833d492ce lrc: 4/1,0 mode: --/PR res: [0x3d7bcf:0x0:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) flags: 0x106400020000 nid: local remote: 0xbe8d0a2d0305e72a expref: -99 pid: 12965 timeout: 0 lvb_type: 1 Aug 16 13:14:01 oak-gw06 kernel: LustreError: 13111:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x3d7bcf:0x0:0x0].0x0 (ffff88019732a9c0) refcount = 2 Aug 16 13:14:01 oak-gw06 kernel: LustreError: 13111:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 16 13:14:01 oak-gw06 kernel: Lustre: oak-OST0009-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 16 13:14:01 oak-gw06 kernel: Lustre: Skipped 2 previous similar messages Aug 16 13:14:04 oak-gw06 kernel: LustreError: 11-0: oak-OST0027-osc-ffff88041b99c000: operation ost_setattr to node 10.0.2.101@o2ib5 failed: rc = -107 Aug 16 13:14:04 oak-gw06 kernel: LustreError: Skipped 6 previous similar messages Aug 16 13:14:05 oak-gw06 kernel: LustreError: 1761:0:(import.c:671:ptlrpc_connect_import()) already connecting Aug 16 13:14:05 oak-gw06 kernel: LustreError: 1761:0:(import.c:671:ptlrpc_connect_import()) Skipped 2 previous similar messages Aug 16 13:14:15 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1502913699/real 1502913699] req@ffff880361b78600 x1566271273452032/t0(0) o400->oak-OST001f-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1502914455 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 16 13:14:15 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 270 previous similar messages Aug 16 13:14:26 oak-gw06 kernel: LustreError: 167-0: oak-OST0005-osc-ffff88041b99c000: This client was evicted by oak-OST0005; in progress operations using this service will fail. Aug 16 13:14:27 oak-gw06 kernel: LustreError: 13117:0:(ldlm_resource.c:882:ldlm_resource_complain()) oak-OST0011-osc-ffff88041b99c000: namespace resource [0x3c8b5f:0x0:0x0].0x0 (ffff880397d96840) refcount nonzero (1) after lock cleanup; forcing cleanup. Aug 16 13:14:27 oak-gw06 kernel: LustreError: 13117:0:(ldlm_resource.c:882:ldlm_resource_complain()) Skipped 1 previous similar message Aug 16 13:14:27 oak-gw06 kernel: LustreError: 13117:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x3c8b5f:0x0:0x0].0x0 (ffff880397d96840) refcount = 2 Aug 16 13:14:27 oak-gw06 kernel: LustreError: 13124:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x1c2acf:0x0:0x0].0x0 (ffff8803e8b3b180) refcount = 3 Aug 16 13:14:27 oak-gw06 kernel: LustreError: 13124:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 16 13:14:27 oak-gw06 kernel: LustreError: 13124:0:(ldlm_resource.c:1486:ldlm_resource_dump()) ### ### ns: oak-OST0027-osc-ffff88041b99c000 lock: ffff8803d53ab800/0xf077f1a833d48d2c lrc: 4/1,0 mode: --/PR res: [0x1c2acf:0x0:0x0].0x0 rrc: 3 type: EXT [0->4194303] (req 0->4194303) flags: 0x106400020000 nid: local remote: 0xbe8d0a2d0305e532 expref: -99 pid: 12950 timeout: 0 lvb_type: 1 Aug 16 13:14:27 oak-gw06 kernel: LustreError: 13124:0:(ldlm_resource.c:1486:ldlm_resource_dump()) Skipped 1 previous similar message Aug 16 13:14:27 oak-gw06 kernel: LustreError: 13124:0:(ldlm_resource.c:1463:ldlm_resource_dump()) --- Resource: [0x1c2acf:0x0:0x0].0x0 (ffff8803e8b3b180) refcount = 3 Aug 16 13:14:27 oak-gw06 kernel: LustreError: 13124:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 16 13:14:27 oak-gw06 kernel: LustreError: 13117:0:(ldlm_resource.c:1484:ldlm_resource_dump()) Waiting locks: Aug 16 13:14:51 oak-gw06 kernel: LustreError: 167-0: oak-OST0001-osc-ffff88041b99c000: This client was evicted by oak-OST0001; in progress operations using this service will fail. Aug 16 13:14:51 oak-gw06 kernel: LustreError: Skipped 12 previous similar messages Aug 16 13:14:51 oak-gw06 kernel: Lustre: 1764:0:(llite_lib.c:2622:ll_dirty_page_discard_warn()) oak: dirty page discard: 10.0.2.51@o2ib5:10.0.2.52@o2ib5:/oak/fid: [0x200002ea3:0x1135b:0x0]// may get corrupted (rc -108) Aug 16 13:14:51 oak-gw06 kernel: Lustre: oak-OST002d-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 16 13:14:51 oak-gw06 kernel: Lustre: Skipped 13 previous similar messages Aug 16 13:41:33 oak-gw06 kernel: Lustre: DEBUG MARKER: Wed Aug 16 13:41:33 2017 Aug 19 07:48:43 oak-gw06 kernel: LustreError: 11-0: oak-OST001f-osc-ffff88041b99c000: operation ldlm_enqueue to node 10.0.2.101@o2ib5 failed: rc = -19 Aug 19 07:48:43 oak-gw06 kernel: LustreError: Skipped 26 previous similar messages Aug 19 07:48:43 oak-gw06 kernel: Lustre: oak-OST001f-osc-ffff88041b99c000: Connection to oak-OST001f (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 19 07:48:43 oak-gw06 kernel: Lustre: Skipped 20 previous similar messages Aug 19 07:49:33 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1503154167/real 1503154167] req@ffff8802a3064300 x1566272738201392/t0(0) o8->oak-OST000d-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1503154173 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 19 07:50:06 oak-gw06 kernel: Lustre: oak-OST0005-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 19 07:50:06 oak-gw06 kernel: Lustre: Skipped 2 previous similar messages Aug 19 07:50:17 oak-gw06 kernel: Lustre: oak-OST0019-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 19 07:50:17 oak-gw06 kernel: Lustre: Skipped 3 previous similar messages Aug 19 07:51:19 oak-gw06 kernel: Lustre: DEBUG MARKER: Sat Aug 19 07:51:19 2017 Aug 19 07:51:26 oak-gw06 kernel: Lustre: oak-OST0003-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 19 07:51:26 oak-gw06 kernel: Lustre: Skipped 16 previous similar messages Aug 21 13:58:19 oak-gw06 kernel: Lustre: 1768:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1503349092/real 1503349092] req@ffff88026e5ea400 x1566273141684912/t0(0) o400->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1503349099 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 21 13:58:19 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1503349092/real 1503349092] req@ffff88026e5e9800 x1566273141684848/t0(0) o400->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1503349099 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 21 13:58:19 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1503349092/real 1503349092] req@ffff88026e5e9e00 x1566273141684880/t0(0) o400->oak-OST0002-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1503349099 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 21 13:58:19 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Aug 21 13:58:19 oak-gw06 kernel: Lustre: 1766:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Aug 21 13:58:19 oak-gw06 kernel: Lustre: oak-OST0000-osc-ffff88041b99c000: Connection to oak-OST0000 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 21 13:58:19 oak-gw06 kernel: Lustre: oak-OST0006-osc-ffff88041b99c000: Connection to oak-OST0006 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 21 13:58:19 oak-gw06 kernel: Lustre: Skipped 24 previous similar messages Aug 21 13:58:19 oak-gw06 kernel: Lustre: Skipped 24 previous similar messages Aug 21 13:58:19 oak-gw06 kernel: Lustre: 1768:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Aug 21 13:58:25 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1503349099/real 1503349099] req@ffff880128e45b00 x1566273141685968/t0(0) o8->oak-OST0024-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1503349105 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 21 13:58:25 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Aug 21 13:59:09 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1503349149/real 1503349149] req@ffff88017703e700 x1566273141686832/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1503349160 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 21 13:59:09 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Aug 21 13:59:50 oak-gw06 kernel: Lustre: oak-OST0010-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 21 13:59:50 oak-gw06 kernel: Lustre: Skipped 2 previous similar messages Aug 21 14:01:01 oak-gw06 kernel: Lustre: oak-OST0026-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 21 18:42:54 oak-gw06 kernel: LustreError: 11-0: oak-OST0000-osc-ffff88041b99c000: operation obd_ping to node 10.0.2.102@o2ib5 failed: rc = -107 Aug 21 18:42:54 oak-gw06 kernel: Lustre: oak-OST000e-osc-ffff88041b99c000: Connection to oak-OST000e (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 21 18:42:54 oak-gw06 kernel: Lustre: Skipped 22 previous similar messages Aug 21 18:42:54 oak-gw06 kernel: LustreError: Skipped 22 previous similar messages Aug 21 18:43:19 oak-gw06 kernel: LustreError: 11-0: oak-OST0003-osc-ffff88041b99c000: operation obd_ping to node 10.0.2.102@o2ib5 failed: rc = -107 Aug 21 18:43:19 oak-gw06 kernel: Lustre: oak-OST0023-osc-ffff88041b99c000: Connection to oak-OST0023 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 21 18:43:19 oak-gw06 kernel: Lustre: Skipped 22 previous similar messages Aug 21 18:43:19 oak-gw06 kernel: LustreError: Skipped 22 previous similar messages Aug 21 18:43:25 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1503366199/real 1503366199] req@ffff88028b7aaa00 x1566273145784560/t0(0) o8->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1503366205 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 21 18:43:25 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Aug 21 18:43:44 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1503366224/real 1503366224] req@ffff8801108d2700 x1566273145785488/t0(0) o400->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1503366231 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 21 18:43:44 oak-gw06 kernel: Lustre: oak-OST001f-osc-ffff88041b99c000: Connection to oak-OST001f (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 21 18:43:44 oak-gw06 kernel: Lustre: Skipped 19 previous similar messages Aug 21 18:43:44 oak-gw06 kernel: Lustre: 1765:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 34 previous similar messages Aug 21 18:43:50 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1503366224/real 1503366224] req@ffff8801c3fd6d00 x1566273145785888/t0(0) o8->oak-OST0013-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1503366230 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 21 18:43:50 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Aug 21 18:44:09 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1503366249/real 1503366249] req@ffff8801c3fd6a00 x1566273145786048/t0(0) o8->oak-OST0000-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1503366260 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 21 18:44:09 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 28 previous similar messages Aug 21 18:44:15 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1503366249/real 1503366249] req@ffff8802b63f6d00 x1566273145786672/t0(0) o8->oak-OST0027-osc-ffff88041b99c000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1503366255 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 21 18:44:15 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 28 previous similar messages Aug 21 18:44:34 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1503366274/real 1503366274] req@ffff88010a9b4600 x1566273145787376/t0(0) o8->oak-OST0004-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1503366290 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 21 18:44:34 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Aug 21 18:44:59 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1503366299/real 1503366299] req@ffff880074a13300 x1566273145788176/t0(0) o8->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1503366315 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 21 18:44:59 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Aug 21 18:45:36 oak-gw06 kernel: Lustre: oak-OST0024-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 21 18:45:36 oak-gw06 kernel: Lustre: Skipped 22 previous similar messages Aug 21 18:45:38 oak-gw06 kernel: Lustre: oak-OST002e-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 21 18:45:38 oak-gw06 kernel: Lustre: Skipped 1 previous similar message Aug 21 18:46:20 oak-gw06 kernel: Lustre: oak-OST0022-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 21 18:46:20 oak-gw06 kernel: Lustre: Skipped 12 previous similar messages Aug 21 18:46:47 oak-gw06 kernel: Lustre: oak-OST0002-osc-ffff88041b99c000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Aug 21 18:46:47 oak-gw06 kernel: Lustre: Skipped 7 previous similar messages Aug 21 19:24:59 oak-gw06 kernel: LustreError: 11-0: oak-OST0017-osc-ffff88041b99c000: operation obd_ping to node 10.0.2.101@o2ib5 failed: rc = -107 Aug 21 19:24:59 oak-gw06 kernel: LustreError: 11-0: oak-OST0007-osc-ffff88041b99c000: operation obd_ping to node 10.0.2.101@o2ib5 failed: rc = -107 Aug 21 19:24:59 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 21 19:24:59 oak-gw06 kernel: Lustre: oak-OST0007-osc-ffff88041b99c000: Connection to oak-OST0007 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Aug 21 19:24:59 oak-gw06 kernel: Lustre: Skipped 3 previous similar messages Aug 21 19:24:59 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Aug 21 19:25:30 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1503368724/real 1503368724] req@ffff880122004c00 x1566273145864784/t0(0) o8->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1503368730 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 21 19:25:30 oak-gw06 kernel: Lustre: 1761:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Aug 21 19:25:54 oak-gw06 kernel: Lustre: oak-OST002f-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 21 19:25:54 oak-gw06 kernel: Lustre: Skipped 24 previous similar messages Aug 21 19:25:56 oak-gw06 kernel: Lustre: oak-OST0007-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 21 19:25:56 oak-gw06 kernel: Lustre: Skipped 6 previous similar messages Aug 21 19:25:59 oak-gw06 kernel: Lustre: oak-OST0011-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 21 19:25:59 oak-gw06 kernel: Lustre: Skipped 3 previous similar messages Aug 21 19:26:20 oak-gw06 kernel: Lustre: oak-OST001d-osc-ffff88041b99c000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Aug 21 19:26:20 oak-gw06 kernel: Lustre: Skipped 7 previous similar messages Aug 23 03:05:34 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 23 03:05:34 oak-gw06 kernel: CPU: 1 PID: 19296 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 23 03:05:34 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 23 03:05:34 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 23 03:05:34 oak-gw06 kernel: 00000000000080d0 00000000215865d7 ffff88030a4ff858 ffffffff8168662f Aug 23 03:05:34 oak-gw06 kernel: ffff88030a4ff8e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 23 03:05:34 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff88030a4ff8e8 00000000215865d7 Aug 23 03:05:34 oak-gw06 kernel: Call Trace: Aug 23 03:05:34 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 23 03:05:34 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 23 03:05:34 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 23 03:05:34 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 23 03:05:34 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 23 03:05:34 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 23 03:05:34 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 23 03:05:34 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 23 03:05:34 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 23 03:05:34 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 23 03:05:34 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 23 03:05:34 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 23 03:05:34 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 23 03:05:34 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 23 03:05:34 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 23 03:05:34 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 23 03:05:34 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 23 03:05:34 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 23 03:05:34 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 23 03:05:34 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 23 03:05:34 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 23 03:05:34 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 23 03:05:34 oak-gw06 kernel: Mem-Info: Aug 23 03:05:34 oak-gw06 kernel: active_anon:12909 inactive_anon:56520 isolated_anon:0#012 active_file:1242433 inactive_file:781962 isolated_file:0#012 unevictable:0 dirty:2509 writeback:608 unstable:0#012 slab_reclaimable:81828 slab_unreclaimable:1461185#012 mapped:13279 shmem:49157 pagetables:1525 bounce:0#012 free:335855 free_pcp:386 free_cma:0 Aug 23 03:05:34 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 23 03:05:35 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 23 03:05:35 oak-gw06 kernel: Node 0 DMA32 free:298984kB min:69724kB low:87152kB high:104584kB active_anon:6900kB inactive_anon:41072kB active_file:856556kB inactive_file:782168kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1828kB writeback:2836kB mapped:9712kB shmem:31268kB slab_reclaimable:28596kB slab_unreclaimable:815460kB kernel_stack:912kB pagetables:1084kB unstable:0kB bounce:0kB free_pcp:268kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 23 03:05:35 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 23 03:05:35 oak-gw06 kernel: Node 0 Normal free:1027736kB min:323104kB low:403880kB high:484656kB active_anon:44736kB inactive_anon:185008kB active_file:4113176kB inactive_file:2348424kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8112kB writeback:0kB mapped:43404kB shmem:165360kB slab_reclaimable:298716kB slab_unreclaimable:5029264kB kernel_stack:4784kB pagetables:5016kB unstable:0kB bounce:0kB free_pcp:1080kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 23 03:05:35 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 23 03:05:35 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 23 03:05:35 oak-gw06 kernel: Node 0 DMA32: 6225*4kB (UEM) 7654*8kB (UEM) 3216*16kB (UEM) 2551*32kB (UEM) 594*64kB (UEM) 207*128kB (UEM) 32*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 291924kB Aug 23 03:05:35 oak-gw06 kernel: Node 0 Normal: 35463*4kB (UEM) 37214*8kB (UEM) 15539*16kB (UEM) 9808*32kB (UEM) 440*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1030204kB Aug 23 03:05:35 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 23 03:05:35 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 23 03:05:35 oak-gw06 kernel: 1986377 total pagecache pages Aug 23 03:05:35 oak-gw06 kernel: 2 pages in swap cache Aug 23 03:05:35 oak-gw06 kernel: Swap cache stats: add 91, delete 89, find 0/0 Aug 23 03:05:35 oak-gw06 kernel: Free swap = 4193936kB Aug 23 03:05:35 oak-gw06 kernel: Total swap = 4194300kB Aug 23 03:05:35 oak-gw06 kernel: 4194203 pages RAM Aug 23 03:05:35 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 23 03:05:35 oak-gw06 kernel: 127313 pages reserved Aug 23 03:05:35 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 23 03:05:35 oak-gw06 kernel: CPU: 1 PID: 19296 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 23 03:05:35 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 23 03:05:35 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 23 03:05:35 oak-gw06 kernel: 00000000000080d0 00000000215865d7 ffff88030a4ff808 ffffffff8168662f Aug 23 03:05:35 oak-gw06 kernel: ffff88030a4ff898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 23 03:05:35 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88030a4ff868 00000000215865d7 Aug 23 03:05:35 oak-gw06 kernel: Call Trace: Aug 23 03:05:35 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 23 03:05:35 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 23 03:05:35 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 23 03:05:35 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 23 03:05:35 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 23 03:05:35 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 23 03:05:35 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 23 03:05:35 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 23 03:05:35 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 23 03:05:35 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 23 03:05:35 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 23 03:05:35 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 23 03:05:35 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 23 03:05:35 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 23 03:05:35 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 23 03:05:35 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 23 03:05:35 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 23 03:05:35 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 23 03:05:35 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 23 03:05:35 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 23 03:05:35 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 23 03:05:35 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 23 03:05:35 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 23 03:05:35 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 23 03:05:35 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 23 03:05:35 oak-gw06 kernel: Mem-Info: Aug 23 03:05:35 oak-gw06 kernel: active_anon:12909 inactive_anon:56520 isolated_anon:0#012 active_file:1242303 inactive_file:785382 isolated_file:0#012 unevictable:0 dirty:2533 writeback:1117 unstable:0#012 slab_reclaimable:81828 slab_unreclaimable:1461185#012 mapped:13279 shmem:49157 pagetables:1525 bounce:0#012 free:332426 free_pcp:539 free_cma:0 Aug 23 03:05:35 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 23 03:05:35 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 23 03:05:35 oak-gw06 kernel: Node 0 DMA32 free:298840kB min:69724kB low:87152kB high:104584kB active_anon:6900kB inactive_anon:41072kB active_file:856556kB inactive_file:785304kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:1244kB writeback:208kB mapped:9712kB shmem:31268kB slab_reclaimable:28596kB slab_unreclaimable:815460kB kernel_stack:912kB pagetables:1084kB unstable:0kB bounce:0kB free_pcp:1184kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 23 03:05:35 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 23 03:05:35 oak-gw06 kernel: Node 0 Normal free:1012032kB min:323104kB low:403880kB high:484656kB active_anon:44736kB inactive_anon:185008kB active_file:4112656kB inactive_file:2359864kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:8500kB writeback:3872kB mapped:43404kB shmem:165360kB slab_reclaimable:298716kB slab_unreclaimable:5029264kB kernel_stack:4784kB pagetables:5016kB unstable:0kB bounce:0kB free_pcp:1292kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 23 03:05:35 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 23 03:05:35 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 23 03:05:35 oak-gw06 kernel: Node 0 DMA32: 6953*4kB (UEM) 7654*8kB (UEM) 3546*16kB (UEM) 2559*32kB (UEM) 594*64kB (UEM) 207*128kB (UEM) 32*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 300372kB Aug 23 03:05:35 oak-gw06 kernel: Node 0 Normal: 31959*4kB (UEM) 37215*8kB (UEM) 15237*16kB (UEM) 9810*32kB (UEM) 440*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1011428kB Aug 23 03:05:35 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 23 03:05:35 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 23 03:05:35 oak-gw06 kernel: 1988850 total pagecache pages Aug 23 03:05:35 oak-gw06 kernel: 2 pages in swap cache Aug 23 03:05:35 oak-gw06 kernel: Swap cache stats: add 91, delete 89, find 0/0 Aug 23 03:05:35 oak-gw06 kernel: Free swap = 4193936kB Aug 23 03:05:35 oak-gw06 kernel: Total swap = 4194300kB Aug 23 03:05:35 oak-gw06 kernel: 4194203 pages RAM Aug 23 03:05:35 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 23 03:05:35 oak-gw06 kernel: 127313 pages reserved Aug 23 03:20:34 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 23 03:20:34 oak-gw06 kernel: CPU: 1 PID: 19461 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 23 03:20:34 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 23 03:20:34 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 23 03:20:34 oak-gw06 kernel: 00000000000080d0 0000000035870738 ffff88014c2bf858 ffffffff8168662f Aug 23 03:20:34 oak-gw06 kernel: ffff88014c2bf8e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 23 03:20:34 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff88014c2bf8e8 0000000035870738 Aug 23 03:20:34 oak-gw06 kernel: Call Trace: Aug 23 03:20:34 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 23 03:20:34 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 23 03:20:34 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 23 03:20:34 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 23 03:20:34 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 23 03:20:34 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 23 03:20:34 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 23 03:20:34 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 23 03:20:34 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 23 03:20:34 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 23 03:20:34 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 23 03:20:34 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 23 03:20:34 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 23 03:20:34 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 23 03:20:34 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 23 03:20:34 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 23 03:20:34 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 23 03:20:34 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 23 03:20:34 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 23 03:20:34 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 23 03:20:34 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 23 03:20:34 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 23 03:20:34 oak-gw06 kernel: Mem-Info: Aug 23 03:20:34 oak-gw06 kernel: active_anon:17430 inactive_anon:56484 isolated_anon:0#012 active_file:1317970 inactive_file:718711 isolated_file:0#012 unevictable:0 dirty:8893 writeback:71 unstable:0#012 slab_reclaimable:81694 slab_unreclaimable:1464883#012 mapped:13303 shmem:49121 pagetables:1720 bounce:0#012 free:316056 free_pcp:7 free_cma:0 Aug 23 03:20:34 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 23 03:20:34 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 23 03:20:34 oak-gw06 kernel: Node 0 DMA32 free:576332kB min:69724kB low:87152kB high:104584kB active_anon:4328kB inactive_anon:41056kB active_file:863080kB inactive_file:555376kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:208kB writeback:0kB mapped:9736kB shmem:31248kB slab_reclaimable:28556kB slab_unreclaimable:788312kB kernel_stack:912kB pagetables:1068kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 23 03:20:34 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 23 03:20:34 oak-gw06 kernel: Node 0 Normal free:671252kB min:323104kB low:403880kB high:484656kB active_anon:65652kB inactive_anon:184880kB active_file:4408800kB inactive_file:2319468kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:35364kB writeback:284kB mapped:43476kB shmem:165236kB slab_reclaimable:298220kB slab_unreclaimable:5071204kB kernel_stack:4800kB pagetables:5812kB unstable:0kB bounce:0kB free_pcp:692kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 23 03:20:34 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 23 03:20:34 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 23 03:20:34 oak-gw06 kernel: Node 0 DMA32: 6257*4kB (UEM) 8201*8kB (UEM) 5103*16kB (UEM) 4633*32kB (UEM) 2015*64kB (UEM) 837*128kB (UEM) 73*256kB (UM) 2*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 576348kB Aug 23 03:20:34 oak-gw06 kernel: Node 0 Normal: 6240*4kB (UEM) 10825*8kB (UEM) 19489*16kB (UEM) 7756*32kB (UEM) 10*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 672216kB Aug 23 03:20:34 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 23 03:20:34 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 23 03:20:34 oak-gw06 kernel: 2085774 total pagecache pages Aug 23 03:20:34 oak-gw06 kernel: 2 pages in swap cache Aug 23 03:20:34 oak-gw06 kernel: Swap cache stats: add 127, delete 125, find 0/0 Aug 23 03:20:34 oak-gw06 kernel: Free swap = 4193792kB Aug 23 03:20:34 oak-gw06 kernel: Total swap = 4194300kB Aug 23 03:20:34 oak-gw06 kernel: 4194203 pages RAM Aug 23 03:20:34 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 23 03:20:34 oak-gw06 kernel: 127313 pages reserved Aug 23 03:20:34 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 23 03:20:34 oak-gw06 kernel: CPU: 1 PID: 19461 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 23 03:20:34 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 23 03:20:34 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 23 03:20:34 oak-gw06 kernel: 00000000000080d0 0000000035870738 ffff88014c2bf808 ffffffff8168662f Aug 23 03:20:34 oak-gw06 kernel: ffff88014c2bf898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 23 03:20:34 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88014c2bf868 0000000035870738 Aug 23 03:20:34 oak-gw06 kernel: Call Trace: Aug 23 03:20:34 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 23 03:20:34 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 23 03:20:34 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 23 03:20:34 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 23 03:20:34 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 23 03:20:34 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 23 03:20:34 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 23 03:20:34 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 23 03:20:34 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 23 03:20:34 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 23 03:20:34 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 23 03:20:34 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 23 03:20:34 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 23 03:20:34 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 23 03:20:34 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 23 03:20:34 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 23 03:20:34 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 23 03:20:34 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 23 03:20:34 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 23 03:20:34 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 23 03:20:34 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 23 03:20:34 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 23 03:20:34 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 23 03:20:34 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 23 03:20:34 oak-gw06 kernel: Mem-Info: Aug 23 03:20:34 oak-gw06 kernel: active_anon:17495 inactive_anon:56484 isolated_anon:0#012 active_file:1317905 inactive_file:718711 isolated_file:0#012 unevictable:0 dirty:8893 writeback:71 unstable:0#012 slab_reclaimable:81694 slab_unreclaimable:1464883#012 mapped:13303 shmem:49121 pagetables:1720 bounce:0#012 free:316194 free_pcp:31 free_cma:0 Aug 23 03:20:34 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 23 03:20:34 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 23 03:20:34 oak-gw06 kernel: Node 0 DMA32 free:576332kB min:69724kB low:87152kB high:104584kB active_anon:4328kB inactive_anon:41056kB active_file:863080kB inactive_file:555376kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:208kB writeback:0kB mapped:9736kB shmem:31248kB slab_reclaimable:28556kB slab_unreclaimable:788312kB kernel_stack:912kB pagetables:1068kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 23 03:20:34 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 23 03:20:34 oak-gw06 kernel: Node 0 Normal free:672552kB min:323104kB low:403880kB high:484656kB active_anon:65652kB inactive_anon:184880kB active_file:4408540kB inactive_file:2319468kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:35364kB writeback:284kB mapped:43476kB shmem:165236kB slab_reclaimable:298220kB slab_unreclaimable:5071204kB kernel_stack:4800kB pagetables:5812kB unstable:0kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 23 03:20:34 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 23 03:20:34 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 23 03:20:34 oak-gw06 kernel: Node 0 DMA32: 6257*4kB (UEM) 8201*8kB (UEM) 5103*16kB (UEM) 4633*32kB (UEM) 2015*64kB (UEM) 837*128kB (UEM) 73*256kB (UM) 2*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 576348kB Aug 23 03:20:34 oak-gw06 kernel: Node 0 Normal: 6233*4kB (UEM) 10832*8kB (UEM) 19515*16kB (UEM) 7757*32kB (UEM) 10*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 672692kB Aug 23 03:20:34 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 23 03:20:34 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 23 03:20:34 oak-gw06 kernel: 2085677 total pagecache pages Aug 23 03:20:34 oak-gw06 kernel: 2 pages in swap cache Aug 23 03:20:34 oak-gw06 kernel: Swap cache stats: add 127, delete 125, find 0/0 Aug 23 03:20:34 oak-gw06 kernel: Free swap = 4193792kB Aug 23 03:20:34 oak-gw06 kernel: Total swap = 4194300kB Aug 23 03:20:34 oak-gw06 kernel: 4194203 pages RAM Aug 23 03:20:35 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 23 03:20:35 oak-gw06 kernel: 127313 pages reserved Aug 24 05:17:28 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 05:17:28 oak-gw06 kernel: CPU: 1 PID: 24041 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:17:28 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:17:28 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:17:28 oak-gw06 kernel: 00000000000080d0 00000000acc8a13a ffff8800831bb858 ffffffff8168662f Aug 24 05:17:28 oak-gw06 kernel: ffff8800831bb8e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 24 05:17:28 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff8800831bb8e8 00000000acc8a13a Aug 24 05:17:28 oak-gw06 kernel: Call Trace: Aug 24 05:17:28 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:17:28 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:17:28 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:17:28 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:17:28 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 05:17:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 05:17:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:17:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:17:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:17:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:17:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:17:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:17:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:17:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:17:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:17:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:17:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:17:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:17:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:17:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:17:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:17:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:17:28 oak-gw06 kernel: Mem-Info: Aug 24 05:17:28 oak-gw06 kernel: active_anon:21297 inactive_anon:57055 isolated_anon:0#012 active_file:259098 inactive_file:1724735 isolated_file:0#012 unevictable:0 dirty:17502 writeback:10537 unstable:0#012 slab_reclaimable:76634 slab_unreclaimable:1409190#012 mapped:13433 shmem:49051 pagetables:1726 bounce:0#012 free:415530 free_pcp:1062 free_cma:0 Aug 24 05:17:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:17:28 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:17:28 oak-gw06 kernel: Node 0 DMA32 free:568560kB min:69724kB low:87152kB high:104584kB active_anon:7544kB inactive_anon:43560kB active_file:180456kB inactive_file:1258132kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:7224kB writeback:7328kB mapped:9312kB shmem:31220kB slab_reclaimable:26408kB slab_unreclaimable:749044kB kernel_stack:1008kB pagetables:1108kB unstable:0kB bounce:0kB free_pcp:1812kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:17:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:17:28 oak-gw06 kernel: Node 0 Normal free:1068348kB min:323104kB low:403880kB high:484656kB active_anon:78164kB inactive_anon:184660kB active_file:855936kB inactive_file:5644708kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:64336kB writeback:37536kB mapped:44420kB shmem:164984kB slab_reclaimable:280128kB slab_unreclaimable:4888244kB kernel_stack:4688kB pagetables:5796kB unstable:0kB bounce:0kB free_pcp:2844kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:17:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:17:28 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:17:28 oak-gw06 kernel: Node 0 DMA32: 4979*4kB (UEM) 6268*8kB (UEM) 8829*16kB (UEM) 7060*32kB (UEM) 1504*64kB (UEM) 223*128kB (UM) 13*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 565372kB Aug 24 05:17:28 oak-gw06 kernel: Node 0 Normal: 10690*4kB (UEM) 11552*8kB (UEM) 21169*16kB (UEM) 11848*32kB (UEM) 2679*64kB (UEM) 255*128kB (UM) 48*256kB (UM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 1069912kB Aug 24 05:17:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:17:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:17:28 oak-gw06 kernel: 1989892 total pagecache pages Aug 24 05:17:28 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:17:28 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:17:28 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:17:28 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:17:28 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:17:28 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:17:28 oak-gw06 kernel: 127313 pages reserved Aug 24 05:17:28 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 05:17:28 oak-gw06 kernel: CPU: 1 PID: 24041 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:17:28 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:17:28 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:17:28 oak-gw06 kernel: 00000000000080d0 00000000acc8a13a ffff8800831bb808 ffffffff8168662f Aug 24 05:17:28 oak-gw06 kernel: ffff8800831bb898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 05:17:28 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800831bb868 00000000acc8a13a Aug 24 05:17:28 oak-gw06 kernel: Call Trace: Aug 24 05:17:28 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:17:28 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:17:28 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:17:28 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:17:28 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 05:17:28 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 05:17:28 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 05:17:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 05:17:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:17:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:17:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:17:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:17:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:17:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:17:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:17:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:17:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:17:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:17:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:17:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:17:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:17:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:17:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:17:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:17:28 oak-gw06 kernel: Mem-Info: Aug 24 05:17:28 oak-gw06 kernel: active_anon:21297 inactive_anon:57055 isolated_anon:0#012 active_file:259098 inactive_file:1728330 isolated_file:0#012 unevictable:0 dirty:18935 writeback:11155 unstable:0#012 slab_reclaimable:76634 slab_unreclaimable:1409582#012 mapped:13433 shmem:49051 pagetables:1726 bounce:0#012 free:411568 free_pcp:629 free_cma:0 Aug 24 05:17:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:17:28 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:17:28 oak-gw06 kernel: Node 0 DMA32 free:564600kB min:69724kB low:87152kB high:104584kB active_anon:7544kB inactive_anon:43560kB active_file:180456kB inactive_file:1261072kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:9852kB writeback:5532kB mapped:9312kB shmem:31220kB slab_reclaimable:26408kB slab_unreclaimable:749252kB kernel_stack:1008kB pagetables:1108kB unstable:0kB bounce:0kB free_pcp:692kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:17:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:17:28 oak-gw06 kernel: Node 0 Normal free:1058864kB min:323104kB low:403880kB high:484656kB active_anon:77644kB inactive_anon:184660kB active_file:855936kB inactive_file:5655108kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:68992kB writeback:39088kB mapped:44420kB shmem:164984kB slab_reclaimable:280128kB slab_unreclaimable:4889604kB kernel_stack:4688kB pagetables:5796kB unstable:0kB bounce:0kB free_pcp:1948kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:17:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:17:28 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:17:28 oak-gw06 kernel: Node 0 DMA32: 5397*4kB (UEM) 6105*8kB (UEM) 8878*16kB (UEM) 7060*32kB (UEM) 1504*64kB (UEM) 223*128kB (UM) 13*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 566524kB Aug 24 05:17:28 oak-gw06 kernel: Node 0 Normal: 10584*4kB (UEM) 10053*8kB (UEM) 20967*16kB (UEM) 11848*32kB (UEM) 2679*64kB (UEM) 255*128kB (UM) 48*256kB (UM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 1054264kB Aug 24 05:17:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:17:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:17:28 oak-gw06 kernel: 1993051 total pagecache pages Aug 24 05:17:28 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:17:28 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:17:28 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:17:28 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:17:28 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:17:28 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:17:28 oak-gw06 kernel: 127313 pages reserved Aug 24 05:22:27 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 05:22:27 oak-gw06 kernel: CPU: 7 PID: 24041 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:22:27 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:22:27 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:22:27 oak-gw06 kernel: 00000000000080d0 00000000acc8a13a ffff8800831bb858 ffffffff8168662f Aug 24 05:22:27 oak-gw06 kernel: ffff8800831bb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 05:22:27 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800831bb8b8 00000000acc8a13a Aug 24 05:22:27 oak-gw06 kernel: Call Trace: Aug 24 05:22:27 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:22:27 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:22:27 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:22:27 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:22:27 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 05:22:27 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 05:22:27 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:22:27 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:22:27 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:22:27 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:22:27 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:22:27 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:22:27 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:22:27 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:22:27 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:22:27 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:22:27 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:22:27 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:22:27 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:22:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:22:27 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:22:27 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:22:27 oak-gw06 kernel: Mem-Info: Aug 24 05:22:27 oak-gw06 kernel: active_anon:23480 inactive_anon:57055 isolated_anon:0#012 active_file:10262 inactive_file:2181701 isolated_file:0#012 unevictable:0 dirty:3412 writeback:9145 unstable:0#012 slab_reclaimable:74429 slab_unreclaimable:1355549#012 mapped:13459 shmem:49051 pagetables:1732 bounce:0#012 free:249702 free_pcp:1063 free_cma:0 Aug 24 05:22:27 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:22:27 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:22:27 oak-gw06 kernel: Node 0 DMA32 free:395744kB min:69724kB low:87152kB high:104584kB active_anon:7544kB inactive_anon:43560kB active_file:20760kB inactive_file:1566524kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2428kB writeback:4040kB mapped:9348kB shmem:31220kB slab_reclaimable:25764kB slab_unreclaimable:755388kB kernel_stack:992kB pagetables:1108kB unstable:0kB bounce:0kB free_pcp:660kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:22:27 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:22:27 oak-gw06 kernel: Node 0 Normal free:574988kB min:323104kB low:403880kB high:484656kB active_anon:86376kB inactive_anon:184660kB active_file:20288kB inactive_file:7166780kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:11220kB writeback:19348kB mapped:44488kB shmem:164984kB slab_reclaimable:271952kB slab_unreclaimable:4667056kB kernel_stack:4704kB pagetables:5820kB unstable:0kB bounce:0kB free_pcp:3740kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:22:27 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:22:27 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:22:27 oak-gw06 kernel: Node 0 DMA32: 3454*4kB (UEM) 6041*8kB (UEM) 5099*16kB (UEM) 4683*32kB (UEM) 1037*64kB (UEM) 182*128kB (UEM) 10*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 385808kB Aug 24 05:22:27 oak-gw06 kernel: Node 0 Normal: 16507*4kB (UEM) 15053*8kB (UEM) 12878*16kB (UEM) 4356*32kB (UEM) 698*64kB (UM) 22*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 579380kB Aug 24 05:22:27 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:22:27 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:22:27 oak-gw06 kernel: 2089419 total pagecache pages Aug 24 05:22:27 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:22:27 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:22:27 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:22:27 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:22:27 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:22:27 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:22:27 oak-gw06 kernel: 127313 pages reserved Aug 24 05:22:27 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 05:22:27 oak-gw06 kernel: CPU: 7 PID: 24041 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:22:27 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:22:27 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:22:27 oak-gw06 kernel: 00000000000080d0 00000000acc8a13a ffff8800831bb808 ffffffff8168662f Aug 24 05:22:27 oak-gw06 kernel: ffff8800831bb898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 05:22:27 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8800831bb868 00000000acc8a13a Aug 24 05:22:27 oak-gw06 kernel: Call Trace: Aug 24 05:22:27 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:22:27 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:22:27 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:22:27 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:22:28 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 05:22:28 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 05:22:28 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 05:22:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 05:22:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:22:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:22:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:22:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:22:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:22:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:22:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:22:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:22:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:22:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:22:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:22:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:22:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:22:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:22:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:22:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:22:28 oak-gw06 kernel: Mem-Info: Aug 24 05:22:28 oak-gw06 kernel: active_anon:23480 inactive_anon:57055 isolated_anon:0#012 active_file:10262 inactive_file:2188009 isolated_file:0#012 unevictable:0 dirty:3509 writeback:5750 unstable:0#012 slab_reclaimable:74429 slab_unreclaimable:1355682#012 mapped:13459 shmem:49051 pagetables:1732 bounce:0#012 free:241678 free_pcp:936 free_cma:0 Aug 24 05:22:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:22:28 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:22:28 oak-gw06 kernel: Node 0 DMA32 free:396180kB min:69724kB low:87152kB high:104584kB active_anon:7544kB inactive_anon:43560kB active_file:20760kB inactive_file:1569044kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:2428kB writeback:4040kB mapped:9348kB shmem:31220kB slab_reclaimable:25764kB slab_unreclaimable:754880kB kernel_stack:992kB pagetables:1108kB unstable:0kB bounce:0kB free_pcp:2108kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:22:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:22:28 oak-gw06 kernel: Node 0 Normal free:547124kB min:323104kB low:403880kB high:484656kB active_anon:86376kB inactive_anon:184660kB active_file:20288kB inactive_file:7190792kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:11220kB writeback:21288kB mapped:44488kB shmem:164984kB slab_reclaimable:271952kB slab_unreclaimable:4667052kB kernel_stack:4704kB pagetables:5820kB unstable:0kB bounce:0kB free_pcp:3128kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:22:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:22:28 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:22:28 oak-gw06 kernel: Node 0 DMA32: 3291*4kB (UEM) 5846*8kB (UEM) 5174*16kB (UEM) 4713*32kB (UEM) 1037*64kB (UEM) 182*128kB (UEM) 10*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 385756kB Aug 24 05:22:28 oak-gw06 kernel: Node 0 Normal: 16434*4kB (UEM) 13530*8kB (UEM) 11873*16kB (UEM) 4390*32kB (UEM) 698*64kB (UM) 22*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 551912kB Aug 24 05:22:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:22:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:22:28 oak-gw06 kernel: 2088873 total pagecache pages Aug 24 05:22:28 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:22:28 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:22:28 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:22:28 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:22:28 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:22:28 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:22:28 oak-gw06 kernel: 127313 pages reserved Aug 24 05:27:28 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 24 05:27:28 oak-gw06 kernel: CPU: 1 PID: 23219 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:27:28 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:27:28 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:27:28 oak-gw06 kernel: 00000000000080d0 0000000013dad8c0 ffff880121ffb858 ffffffff8168662f Aug 24 05:27:28 oak-gw06 kernel: ffff880121ffb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 05:27:28 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880121ffb8b8 0000000013dad8c0 Aug 24 05:27:28 oak-gw06 kernel: Call Trace: Aug 24 05:27:28 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:27:28 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:27:28 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:27:28 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:27:28 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 05:27:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 05:27:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:27:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:27:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:27:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:27:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:27:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:27:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:27:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:27:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:27:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:27:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:27:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:27:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:27:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:27:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:27:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:27:28 oak-gw06 kernel: Mem-Info: Aug 24 05:27:28 oak-gw06 kernel: active_anon:23480 inactive_anon:57055 isolated_anon:0#012 active_file:12027 inactive_file:2053572 isolated_file:0#012 unevictable:0 dirty:573 writeback:0 unstable:0#012 slab_reclaimable:73675 slab_unreclaimable:1311090#012 mapped:13471 shmem:49051 pagetables:1732 bounce:0#012 free:442020 free_pcp:190 free_cma:0 Aug 24 05:27:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:27:28 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:27:28 oak-gw06 kernel: Node 0 DMA32 free:491708kB min:69724kB low:87152kB high:104584kB active_anon:7544kB inactive_anon:43560kB active_file:23080kB inactive_file:1499764kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:60kB writeback:0kB mapped:9352kB shmem:31220kB slab_reclaimable:25468kB slab_unreclaimable:735384kB kernel_stack:992kB pagetables:1108kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:27:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:27:28 oak-gw06 kernel: Node 0 Normal free:1257640kB min:323104kB low:403880kB high:484656kB active_anon:86376kB inactive_anon:184660kB active_file:25028kB inactive_file:6716516kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:2232kB writeback:0kB mapped:44544kB shmem:164984kB slab_reclaimable:269232kB slab_unreclaimable:4508720kB kernel_stack:4704kB pagetables:5820kB unstable:0kB bounce:0kB free_pcp:1548kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:27:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:27:28 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:27:28 oak-gw06 kernel: Node 0 DMA32: 5339*4kB (UEM) 5404*8kB (UEM) 6210*16kB (UEM) 5715*32kB (UEM) 1751*64kB (UEM) 249*128kB (UEM) 10*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 493324kB Aug 24 05:27:28 oak-gw06 kernel: Node 0 Normal: 29180*4kB (UE) 18534*8kB (UEM) 19327*16kB (UEM) 17614*32kB (UEM) 1354*64kB (UEM) 130*128kB (UM) 7*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1242960kB Aug 24 05:27:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:27:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:27:28 oak-gw06 kernel: 2024339 total pagecache pages Aug 24 05:27:28 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:27:28 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:27:28 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:27:28 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:27:28 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:27:28 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:27:28 oak-gw06 kernel: 127313 pages reserved Aug 24 05:27:28 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 24 05:27:28 oak-gw06 kernel: CPU: 1 PID: 23219 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:27:28 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:27:28 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:27:28 oak-gw06 kernel: 00000000000080d0 0000000013dad8c0 ffff880121ffb808 ffffffff8168662f Aug 24 05:27:28 oak-gw06 kernel: ffff880121ffb898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 24 05:27:28 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880121ffb868 0000000013dad8c0 Aug 24 05:27:28 oak-gw06 kernel: Call Trace: Aug 24 05:27:28 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:27:28 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:27:28 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 24 05:27:28 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:27:28 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:27:28 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 05:27:28 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 05:27:28 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 05:27:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 05:27:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:27:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:27:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:27:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:27:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:27:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:27:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:27:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:27:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:27:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:27:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:27:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:27:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:27:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:27:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:27:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:27:28 oak-gw06 kernel: Mem-Info: Aug 24 05:27:28 oak-gw06 kernel: active_anon:23480 inactive_anon:57055 isolated_anon:0#012 active_file:12027 inactive_file:2057825 isolated_file:0#012 unevictable:0 dirty:573 writeback:0 unstable:0#012 slab_reclaimable:73675 slab_unreclaimable:1310926#012 mapped:13474 shmem:49051 pagetables:1732 bounce:0#012 free:438306 free_pcp:177 free_cma:0 Aug 24 05:27:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:27:28 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:27:28 oak-gw06 kernel: Node 0 DMA32 free:493324kB min:69724kB low:87152kB high:104584kB active_anon:7544kB inactive_anon:43560kB active_file:23080kB inactive_file:1499880kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:60kB writeback:0kB mapped:9352kB shmem:31220kB slab_reclaimable:25468kB slab_unreclaimable:734984kB kernel_stack:992kB pagetables:1108kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:27:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:27:28 oak-gw06 kernel: Node 0 Normal free:1243996kB min:323104kB low:403880kB high:484656kB active_anon:86896kB inactive_anon:184660kB active_file:25028kB inactive_file:6731420kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:2232kB writeback:0kB mapped:44544kB shmem:164984kB slab_reclaimable:269232kB slab_unreclaimable:4508440kB kernel_stack:4704kB pagetables:5820kB unstable:0kB bounce:0kB free_pcp:976kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:27:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:27:28 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:27:28 oak-gw06 kernel: Node 0 DMA32: 5340*4kB (UEM) 5404*8kB (UEM) 6210*16kB (UEM) 5715*32kB (UEM) 1751*64kB (UEM) 249*128kB (UEM) 10*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 493328kB Aug 24 05:27:28 oak-gw06 kernel: Node 0 Normal: 29233*4kB (UEM) 18650*8kB (UEM) 19345*16kB (UEM) 17615*32kB (UEM) 1354*64kB (UEM) 130*128kB (UM) 7*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1244420kB Aug 24 05:27:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:27:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:27:28 oak-gw06 kernel: 2023983 total pagecache pages Aug 24 05:27:28 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:27:28 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:27:28 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:27:28 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:27:28 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:27:28 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:27:28 oak-gw06 kernel: 127313 pages reserved Aug 24 05:32:28 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 24 05:32:28 oak-gw06 kernel: CPU: 1 PID: 24051 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:32:28 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:32:28 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:32:28 oak-gw06 kernel: 00000000000080d0 00000000bd6af91c ffff8801fc9bb858 ffffffff8168662f Aug 24 05:32:28 oak-gw06 kernel: ffff8801fc9bb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 05:32:28 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801fc9bb8b8 00000000bd6af91c Aug 24 05:32:28 oak-gw06 kernel: Call Trace: Aug 24 05:32:28 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:32:28 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:32:28 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:32:28 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:32:28 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 05:32:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 05:32:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:32:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:32:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:32:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:32:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:32:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:32:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:32:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:32:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:32:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:32:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:32:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:32:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:32:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:32:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:32:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:32:28 oak-gw06 kernel: Mem-Info: Aug 24 05:32:28 oak-gw06 kernel: active_anon:23480 inactive_anon:57055 isolated_anon:0#012 active_file:12150 inactive_file:2147874 isolated_file:0#012 unevictable:0 dirty:12392 writeback:5389 unstable:0#012 slab_reclaimable:73572 slab_unreclaimable:1306905#012 mapped:13482 shmem:49051 pagetables:1732 bounce:0#012 free:318657 free_pcp:1062 free_cma:0 Aug 24 05:32:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:32:28 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:32:28 oak-gw06 kernel: Node 0 DMA32 free:431576kB min:69724kB low:87152kB high:104584kB active_anon:7544kB inactive_anon:43560kB active_file:20980kB inactive_file:1547876kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:13360kB writeback:2424kB mapped:9348kB shmem:31220kB slab_reclaimable:25404kB slab_unreclaimable:727424kB kernel_stack:1008kB pagetables:1108kB unstable:0kB bounce:0kB free_pcp:3068kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:32:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:32:28 oak-gw06 kernel: Node 0 Normal free:816404kB min:323104kB low:403880kB high:484656kB active_anon:86896kB inactive_anon:184660kB active_file:27620kB inactive_file:7050120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:35480kB writeback:18720kB mapped:44580kB shmem:164984kB slab_reclaimable:268884kB slab_unreclaimable:4500988kB kernel_stack:4704kB pagetables:5820kB unstable:0kB bounce:0kB free_pcp:2164kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:32:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:32:28 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:32:28 oak-gw06 kernel: Node 0 DMA32: 3233*4kB (UEM) 9066*8kB (UEM) 7220*16kB (UEM) 3443*32kB (UEM) 1390*64kB (UEM) 271*128kB (UEM) 10*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 437364kB Aug 24 05:32:28 oak-gw06 kernel: Node 0 Normal: 16754*4kB (UE) 28524*8kB (UEM) 12563*16kB (UEM) 7136*32kB (UEM) 1248*64kB (UEM) 55*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 811480kB Aug 24 05:32:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:32:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:32:28 oak-gw06 kernel: 2080022 total pagecache pages Aug 24 05:32:28 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:32:28 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:32:28 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:32:28 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:32:28 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:32:28 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:32:28 oak-gw06 kernel: 127313 pages reserved Aug 24 05:32:28 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 24 05:32:28 oak-gw06 kernel: CPU: 1 PID: 24051 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:32:28 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:32:28 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:32:28 oak-gw06 kernel: 00000000000080d0 00000000bd6af91c ffff8801fc9bb808 ffffffff8168662f Aug 24 05:32:28 oak-gw06 kernel: ffff8801fc9bb898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 24 05:32:28 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801fc9bb868 00000000bd6af91c Aug 24 05:32:28 oak-gw06 kernel: Call Trace: Aug 24 05:32:28 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:32:28 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:32:28 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 24 05:32:28 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:32:28 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:32:28 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 05:32:28 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 05:32:28 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 05:32:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 05:32:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:32:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:32:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:32:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:32:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:32:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:32:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:32:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:32:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:32:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:32:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:32:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:32:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:32:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:32:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:32:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:32:28 oak-gw06 kernel: Mem-Info: Aug 24 05:32:28 oak-gw06 kernel: active_anon:23480 inactive_anon:57055 isolated_anon:0#012 active_file:12150 inactive_file:2154439 isolated_file:0#012 unevictable:0 dirty:12501 writeback:5577 unstable:0#012 slab_reclaimable:73572 slab_unreclaimable:1307651#012 mapped:13482 shmem:49051 pagetables:1732 bounce:0#012 free:312674 free_pcp:482 free_cma:0 Aug 24 05:32:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:32:28 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:32:28 oak-gw06 kernel: Node 0 DMA32 free:438648kB min:69724kB low:87152kB high:104584kB active_anon:7544kB inactive_anon:43560kB active_file:20980kB inactive_file:1547876kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:13360kB writeback:2424kB mapped:9348kB shmem:31220kB slab_reclaimable:25404kB slab_unreclaimable:727424kB kernel_stack:1008kB pagetables:1108kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:32:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:32:28 oak-gw06 kernel: Node 0 Normal free:792212kB min:323104kB low:403880kB high:484656kB active_anon:86896kB inactive_anon:184660kB active_file:27620kB inactive_file:7073000kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:36644kB writeback:17556kB mapped:44580kB shmem:164984kB slab_reclaimable:268884kB slab_unreclaimable:4503708kB kernel_stack:4704kB pagetables:5820kB unstable:0kB bounce:0kB free_pcp:1968kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:32:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:32:28 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:32:28 oak-gw06 kernel: Node 0 DMA32: 4039*4kB (UEM) 9067*8kB (UEM) 7255*16kB (UEM) 3443*32kB (UEM) 1390*64kB (UEM) 271*128kB (UEM) 10*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 441156kB Aug 24 05:32:28 oak-gw06 kernel: Node 0 Normal: 16994*4kB (UEM) 25989*8kB (UEM) 12394*16kB (UEM) 7136*32kB (UEM) 1248*64kB (UEM) 55*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 789456kB Aug 24 05:32:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:32:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:32:28 oak-gw06 kernel: 2084969 total pagecache pages Aug 24 05:32:28 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:32:28 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:32:28 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:32:28 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:32:28 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:32:28 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:32:28 oak-gw06 kernel: 127313 pages reserved Aug 24 05:42:28 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 24 05:42:28 oak-gw06 kernel: CPU: 1 PID: 23219 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:42:28 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:42:28 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:42:28 oak-gw06 kernel: 00000000000080d0 0000000013dad8c0 ffff880121ffb858 ffffffff8168662f Aug 24 05:42:28 oak-gw06 kernel: ffff880121ffb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 05:42:28 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880121ffb8b8 0000000013dad8c0 Aug 24 05:42:28 oak-gw06 kernel: Call Trace: Aug 24 05:42:28 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:42:28 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:42:28 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:42:28 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:42:28 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 05:42:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 05:42:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:42:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:42:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:42:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:42:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:42:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:42:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:42:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:42:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:42:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:42:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:42:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:42:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:42:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:42:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:42:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:42:28 oak-gw06 kernel: Mem-Info: Aug 24 05:42:28 oak-gw06 kernel: active_anon:19597 inactive_anon:57055 isolated_anon:0#012 active_file:12847 inactive_file:2156185 isolated_file:0#012 unevictable:0 dirty:795 writeback:1506 unstable:0#012 slab_reclaimable:72967 slab_unreclaimable:1260415#012 mapped:13506 shmem:49051 pagetables:1720 bounce:0#012 free:392663 free_pcp:951 free_cma:0 Aug 24 05:42:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:42:28 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:42:28 oak-gw06 kernel: Node 0 DMA32 free:451436kB min:69724kB low:87152kB high:104584kB active_anon:7752kB inactive_anon:43560kB active_file:22616kB inactive_file:1627008kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4kB writeback:208kB mapped:9368kB shmem:31220kB slab_reclaimable:25160kB slab_unreclaimable:655032kB kernel_stack:1056kB pagetables:1888kB unstable:0kB bounce:0kB free_pcp:440kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:42:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:42:28 oak-gw06 kernel: Node 0 Normal free:1089348kB min:323104kB low:403880kB high:484656kB active_anon:70636kB inactive_anon:184660kB active_file:28772kB inactive_file:7009432kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:2788kB writeback:5428kB mapped:44656kB shmem:164984kB slab_reclaimable:266708kB slab_unreclaimable:4387428kB kernel_stack:4640kB pagetables:4992kB unstable:0kB bounce:0kB free_pcp:3756kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:42:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:42:28 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:42:28 oak-gw06 kernel: Node 0 DMA32: 5785*4kB (UEM) 7314*8kB (UEM) 1959*16kB (UEM) 5871*32kB (UEM) 1868*64kB (UEM) 248*128kB (UEM) 13*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 456004kB Aug 24 05:42:28 oak-gw06 kernel: Node 0 Normal: 35758*4kB (UEM) 33596*8kB (UEM) 16534*16kB (UEM) 11584*32kB (UEM) 1323*64kB (UEM) 104*128kB (UM) 3*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1145784kB Aug 24 05:42:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:42:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:42:28 oak-gw06 kernel: 2088840 total pagecache pages Aug 24 05:42:28 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:42:28 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:42:28 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:42:28 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:42:28 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:42:28 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:42:28 oak-gw06 kernel: 127313 pages reserved Aug 24 05:47:28 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 24 05:47:28 oak-gw06 kernel: CPU: 1 PID: 24051 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:47:28 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:47:28 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:47:28 oak-gw06 kernel: 00000000000080d0 00000000bd6af91c ffff8801fc9bb858 ffffffff8168662f Aug 24 05:47:28 oak-gw06 kernel: ffff8801fc9bb8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 05:47:28 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff8801fc9bb8b8 00000000bd6af91c Aug 24 05:47:28 oak-gw06 kernel: Call Trace: Aug 24 05:47:28 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:47:28 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:47:28 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:47:28 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:47:28 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 05:47:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 05:47:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:47:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:47:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:47:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:47:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:47:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:47:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:47:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:47:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:47:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:47:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:47:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:47:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:47:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:47:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:47:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:47:28 oak-gw06 kernel: Mem-Info: Aug 24 05:47:28 oak-gw06 kernel: active_anon:16879 inactive_anon:57055 isolated_anon:0#012 active_file:299292 inactive_file:1686392 isolated_file:0#012 unevictable:0 dirty:8400 writeback:962 unstable:0#012 slab_reclaimable:72711 slab_unreclaimable:1243121#012 mapped:13503 shmem:49051 pagetables:1690 bounce:0#012 free:590415 free_pcp:183 free_cma:0 Aug 24 05:47:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:47:28 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:47:28 oak-gw06 kernel: Node 0 DMA32 free:807132kB min:69724kB low:87152kB high:104584kB active_anon:3476kB inactive_anon:43560kB active_file:82248kB inactive_file:1290840kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:812kB writeback:0kB mapped:9324kB shmem:31220kB slab_reclaimable:25084kB slab_unreclaimable:604012kB kernel_stack:960kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:47:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:47:28 oak-gw06 kernel: Node 0 Normal free:1537936kB min:323104kB low:403880kB high:484656kB active_anon:64300kB inactive_anon:184660kB active_file:1114920kB inactive_file:5454728kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:32788kB writeback:3848kB mapped:44688kB shmem:164984kB slab_reclaimable:265760kB slab_unreclaimable:4368456kB kernel_stack:4720kB pagetables:5624kB unstable:0kB bounce:0kB free_pcp:736kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:47:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:47:28 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:47:28 oak-gw06 kernel: Node 0 DMA32: 15793*4kB (UEM) 19176*8kB (UEM) 12946*16kB (UEM) 7065*32kB (UEM) 1947*64kB (UEM) 249*128kB (UEM) 8*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 808324kB Aug 24 05:47:28 oak-gw06 kernel: Node 0 Normal: 46756*4kB (UEM) 37793*8kB (UEM) 34432*16kB (UEM) 12196*32kB (UEM) 1439*64kB (UEM) 107*128kB (UM) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1537880kB Aug 24 05:47:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:47:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:47:28 oak-gw06 kernel: 2034544 total pagecache pages Aug 24 05:47:28 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:47:28 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:47:28 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:47:28 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:47:28 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:47:28 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:47:28 oak-gw06 kernel: 127313 pages reserved Aug 24 05:47:29 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 24 05:47:29 oak-gw06 kernel: CPU: 1 PID: 24051 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:47:29 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:47:29 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:47:29 oak-gw06 kernel: 00000000000080d0 00000000bd6af91c ffff8801fc9bb808 ffffffff8168662f Aug 24 05:47:29 oak-gw06 kernel: ffff8801fc9bb898 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 24 05:47:29 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff8801fc9bb898 00000000bd6af91c Aug 24 05:47:29 oak-gw06 kernel: Call Trace: Aug 24 05:47:29 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:47:29 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:47:29 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:47:29 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:47:29 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 05:47:29 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 05:47:29 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 05:47:29 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 05:47:29 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:47:29 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:47:29 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:47:29 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:47:29 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:47:29 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:47:29 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:47:29 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:47:29 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:47:29 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:47:29 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:47:29 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:47:29 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:47:29 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:47:29 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:47:29 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:47:29 oak-gw06 kernel: Mem-Info: Aug 24 05:47:29 oak-gw06 kernel: active_anon:16697 inactive_anon:57055 isolated_anon:0#012 active_file:300520 inactive_file:1685929 isolated_file:3#012 unevictable:0 dirty:8447 writeback:1473 unstable:0#012 slab_reclaimable:72711 slab_unreclaimable:1242976#012 mapped:13511 shmem:49051 pagetables:1690 bounce:0#012 free:588505 free_pcp:165 free_cma:0 Aug 24 05:47:29 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:47:29 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:47:29 oak-gw06 kernel: Node 0 DMA32 free:808320kB min:69724kB low:87152kB high:104584kB active_anon:3476kB inactive_anon:43560kB active_file:82248kB inactive_file:1290496kB unevictable:0kB isolated(anon):0kB isolated(file):12kB present:3129332kB managed:2884592kB mlocked:0kB dirty:840kB writeback:0kB mapped:9324kB shmem:31220kB slab_reclaimable:25084kB slab_unreclaimable:603980kB kernel_stack:960kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:47:29 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:47:29 oak-gw06 kernel: Node 0 Normal free:1527924kB min:323104kB low:403880kB high:484656kB active_anon:64352kB inactive_anon:184660kB active_file:1119832kB inactive_file:5454000kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:33724kB writeback:5116kB mapped:44720kB shmem:164984kB slab_reclaimable:265760kB slab_unreclaimable:4367908kB kernel_stack:4736kB pagetables:5624kB unstable:0kB bounce:0kB free_pcp:1288kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:47:29 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:47:29 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:47:29 oak-gw06 kernel: Node 0 DMA32: 15794*4kB (UEM) 19177*8kB (UEM) 12945*16kB (UEM) 7065*32kB (UEM) 1947*64kB (UEM) 249*128kB (UEM) 8*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 808320kB Aug 24 05:47:29 oak-gw06 kernel: Node 0 Normal: 44441*4kB (UEM) 38096*8kB (UEM) 34238*16kB (UEM) 12139*32kB (UEM) 1429*64kB (UEM) 107*128kB (UM) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1525476kB Aug 24 05:47:29 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:47:29 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:47:29 oak-gw06 kernel: 2036093 total pagecache pages Aug 24 05:47:29 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:47:29 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:47:29 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:47:29 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:47:29 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:47:29 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:47:29 oak-gw06 kernel: 127313 pages reserved Aug 24 05:52:28 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 05:52:28 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:52:28 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:52:28 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:52:28 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97858 ffffffff8168662f Aug 24 05:52:28 oak-gw06 kernel: ffff880172a978e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 05:52:28 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a978b8 00000000f1450c91 Aug 24 05:52:28 oak-gw06 kernel: Call Trace: Aug 24 05:52:28 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:52:28 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:52:28 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:52:28 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:52:28 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 05:52:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 05:52:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:52:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:52:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:52:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:52:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:52:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:52:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:52:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:52:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:52:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:52:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:52:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:52:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:52:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:52:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:52:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:52:28 oak-gw06 kernel: Mem-Info: Aug 24 05:52:28 oak-gw06 kernel: active_anon:11268 inactive_anon:57055 isolated_anon:0#012 active_file:2024316 inactive_file:4807 isolated_file:0#012 unevictable:0 dirty:49356 writeback:0 unstable:0#012 slab_reclaimable:72653 slab_unreclaimable:1241740#012 mapped:13259 shmem:49051 pagetables:1640 bounce:0#012 free:561291 free_pcp:383 free_cma:0 Aug 24 05:52:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:52:28 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:52:28 oak-gw06 kernel: Node 0 DMA32 free:1019676kB min:69724kB low:87152kB high:104584kB active_anon:3476kB inactive_anon:43560kB active_file:1158032kB inactive_file:10236kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:15776kB writeback:0kB mapped:9204kB shmem:31220kB slab_reclaimable:25056kB slab_unreclaimable:594696kB kernel_stack:960kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:52:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:52:28 oak-gw06 kernel: Node 0 Normal free:1207088kB min:323104kB low:403880kB high:484656kB active_anon:43676kB inactive_anon:184660kB active_file:6939232kB inactive_file:8992kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:181648kB writeback:0kB mapped:43832kB shmem:164984kB slab_reclaimable:265556kB slab_unreclaimable:4372248kB kernel_stack:4752kB pagetables:5424kB unstable:0kB bounce:0kB free_pcp:1960kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:52:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:52:28 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:52:28 oak-gw06 kernel: Node 0 DMA32: 8205*4kB (UEM) 6815*8kB (UEM) 4761*16kB (UEM) 15459*32kB (UEM) 4532*64kB (UEM) 530*128kB (UEM) 14*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1019676kB Aug 24 05:52:28 oak-gw06 kernel: Node 0 Normal: 40389*4kB (UEM) 26366*8kB (UEM) 33194*16kB (UEM) 7160*32kB (UEM) 909*64kB (UEM) 115*128kB (UM) 5*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1206884kB Aug 24 05:52:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:52:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:52:28 oak-gw06 kernel: 2078154 total pagecache pages Aug 24 05:52:28 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:52:28 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:52:28 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:52:28 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:52:28 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:52:28 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:52:28 oak-gw06 kernel: 127313 pages reserved Aug 24 05:52:28 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 05:52:28 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:52:28 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:52:28 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:52:28 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97808 ffffffff8168662f Aug 24 05:52:28 oak-gw06 kernel: ffff880172a97898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 24 05:52:28 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a97868 00000000f1450c91 Aug 24 05:52:28 oak-gw06 kernel: Call Trace: Aug 24 05:52:28 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:52:28 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:52:28 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 24 05:52:28 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:52:28 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:52:28 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 05:52:28 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 05:52:28 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 05:52:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 05:52:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:52:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:52:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:52:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:52:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:52:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:52:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:52:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:52:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:52:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:52:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:52:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:52:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:52:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:52:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:52:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:52:28 oak-gw06 kernel: Mem-Info: Aug 24 05:52:28 oak-gw06 kernel: active_anon:11788 inactive_anon:57055 isolated_anon:0#012 active_file:2024190 inactive_file:4807 isolated_file:0#012 unevictable:0 dirty:49356 writeback:0 unstable:0#012 slab_reclaimable:72653 slab_unreclaimable:1241740#012 mapped:13259 shmem:49051 pagetables:1640 bounce:0#012 free:560859 free_pcp:159 free_cma:0 Aug 24 05:52:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:52:28 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:52:28 oak-gw06 kernel: Node 0 DMA32 free:1019676kB min:69724kB low:87152kB high:104584kB active_anon:3476kB inactive_anon:43560kB active_file:1157528kB inactive_file:10236kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:15776kB writeback:0kB mapped:9204kB shmem:31220kB slab_reclaimable:25056kB slab_unreclaimable:594696kB kernel_stack:960kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:512kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:52:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:52:28 oak-gw06 kernel: Node 0 Normal free:1207220kB min:323104kB low:403880kB high:484656kB active_anon:43936kB inactive_anon:184660kB active_file:6939232kB inactive_file:8992kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:181648kB writeback:0kB mapped:43832kB shmem:164984kB slab_reclaimable:265556kB slab_unreclaimable:4372248kB kernel_stack:4752kB pagetables:5424kB unstable:0kB bounce:0kB free_pcp:316kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:52:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:52:28 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:52:28 oak-gw06 kernel: Node 0 DMA32: 8205*4kB (UEM) 6815*8kB (UEM) 4761*16kB (UEM) 15459*32kB (UEM) 4532*64kB (UEM) 530*128kB (UEM) 14*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1019676kB Aug 24 05:52:28 oak-gw06 kernel: Node 0 Normal: 40592*4kB (UEM) 26366*8kB (UEM) 33194*16kB (UEM) 7159*32kB (UEM) 910*64kB (UEM) 115*128kB (UM) 5*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1207728kB Aug 24 05:52:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:52:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:52:28 oak-gw06 kernel: 2077966 total pagecache pages Aug 24 05:52:28 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:52:28 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:52:28 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:52:28 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:52:28 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:52:28 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:52:28 oak-gw06 kernel: 127313 pages reserved Aug 24 05:57:28 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 05:57:28 oak-gw06 kernel: CPU: 2 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:57:28 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:57:28 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:57:28 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97858 ffffffff8168662f Aug 24 05:57:28 oak-gw06 kernel: ffff880172a978e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 05:57:28 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a978b8 00000000f1450c91 Aug 24 05:57:28 oak-gw06 kernel: Call Trace: Aug 24 05:57:28 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:57:28 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:57:28 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:57:28 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:57:28 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 05:57:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 05:57:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:57:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:57:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:57:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:57:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:57:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:57:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:57:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:57:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:57:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:57:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:57:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:57:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:57:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:57:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:57:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:57:28 oak-gw06 kernel: Mem-Info: Aug 24 05:57:28 oak-gw06 kernel: active_anon:17460 inactive_anon:57055 isolated_anon:0#012 active_file:1904151 inactive_file:27465 isolated_file:0#012 unevictable:0 dirty:3737 writeback:2102 unstable:0#012 slab_reclaimable:72607 slab_unreclaimable:1227392#012 mapped:13521 shmem:49051 pagetables:1710 bounce:0#012 free:660860 free_pcp:284 free_cma:0 Aug 24 05:57:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:57:28 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:57:28 oak-gw06 kernel: Node 0 DMA32 free:1755184kB min:69724kB low:87152kB high:104584kB active_anon:3476kB inactive_anon:43560kB active_file:462168kB inactive_file:7640kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4kB writeback:0kB mapped:9364kB shmem:31220kB slab_reclaimable:25020kB slab_unreclaimable:560212kB kernel_stack:960kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:57:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:57:28 oak-gw06 kernel: Node 0 Normal free:872364kB min:323104kB low:403880kB high:484656kB active_anon:66364kB inactive_anon:184660kB active_file:7154436kB inactive_file:102220kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:14944kB writeback:8408kB mapped:44720kB shmem:164984kB slab_reclaimable:265408kB slab_unreclaimable:4349340kB kernel_stack:4736kB pagetables:5704kB unstable:0kB bounce:0kB free_pcp:1120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:57:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:57:28 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:57:28 oak-gw06 kernel: Node 0 DMA32: 23898*4kB (UEM) 19425*8kB (UEM) 25708*16kB (UEM) 17280*32kB (UEM) 6192*64kB (UEM) 1020*128kB (UEM) 51*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1755184kB Aug 24 05:57:28 oak-gw06 kernel: Node 0 Normal: 29763*4kB (UEM) 32229*8kB (UEM) 18842*16kB (UEM) 3863*32kB (UEM) 810*64kB (UEM) 126*128kB (UM) 8*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 871988kB Aug 24 05:57:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:57:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:57:28 oak-gw06 kernel: 1980582 total pagecache pages Aug 24 05:57:28 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:57:28 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:57:28 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:57:28 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:57:28 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:57:28 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:57:28 oak-gw06 kernel: 127313 pages reserved Aug 24 05:57:28 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 05:57:28 oak-gw06 kernel: CPU: 0 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 05:57:28 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 05:57:28 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 05:57:28 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97808 ffffffff8168662f Aug 24 05:57:28 oak-gw06 kernel: ffff880172a97898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 05:57:28 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a97868 00000000f1450c91 Aug 24 05:57:28 oak-gw06 kernel: Call Trace: Aug 24 05:57:28 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 05:57:28 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 05:57:28 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 05:57:28 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 05:57:28 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 05:57:28 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 05:57:28 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 05:57:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 05:57:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 05:57:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 05:57:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 05:57:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 05:57:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 05:57:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 05:57:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 05:57:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 05:57:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 05:57:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 05:57:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 05:57:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 05:57:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 05:57:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:57:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 05:57:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 05:57:28 oak-gw06 kernel: Mem-Info: Aug 24 05:57:28 oak-gw06 kernel: active_anon:17525 inactive_anon:57055 isolated_anon:0#012 active_file:1904151 inactive_file:27465 isolated_file:0#012 unevictable:0 dirty:3737 writeback:2102 unstable:0#012 slab_reclaimable:72607 slab_unreclaimable:1227392#012 mapped:13521 shmem:49051 pagetables:1710 bounce:0#012 free:660781 free_pcp:54 free_cma:0 Aug 24 05:57:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 05:57:28 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 05:57:28 oak-gw06 kernel: Node 0 DMA32 free:1755184kB min:69724kB low:87152kB high:104584kB active_anon:3476kB inactive_anon:43560kB active_file:462168kB inactive_file:7640kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:4kB writeback:0kB mapped:9364kB shmem:31220kB slab_reclaimable:25020kB slab_unreclaimable:560212kB kernel_stack:960kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:57:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 05:57:28 oak-gw06 kernel: Node 0 Normal free:869176kB min:323104kB low:403880kB high:484656kB active_anon:66624kB inactive_anon:184660kB active_file:7154436kB inactive_file:102220kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:14944kB writeback:8408kB mapped:44720kB shmem:164984kB slab_reclaimable:265408kB slab_unreclaimable:4349340kB kernel_stack:4736kB pagetables:5704kB unstable:0kB bounce:0kB free_pcp:460kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 05:57:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 05:57:28 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 05:57:28 oak-gw06 kernel: Node 0 DMA32: 23898*4kB (UEM) 19425*8kB (UEM) 25708*16kB (UEM) 17280*32kB (UEM) 6192*64kB (UEM) 1020*128kB (UEM) 51*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1755184kB Aug 24 05:57:28 oak-gw06 kernel: Node 0 Normal: 29233*4kB (UEM) 32340*8kB (UEM) 18622*16kB (UEM) 3863*32kB (UEM) 810*64kB (UEM) 126*128kB (UM) 8*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 867236kB Aug 24 05:57:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 05:57:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 05:57:28 oak-gw06 kernel: 1980485 total pagecache pages Aug 24 05:57:28 oak-gw06 kernel: 6 pages in swap cache Aug 24 05:57:28 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 05:57:28 oak-gw06 kernel: Free swap = 4193544kB Aug 24 05:57:28 oak-gw06 kernel: Total swap = 4194300kB Aug 24 05:57:28 oak-gw06 kernel: 4194203 pages RAM Aug 24 05:57:28 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 05:57:28 oak-gw06 kernel: 127313 pages reserved Aug 24 06:02:28 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 24 06:02:28 oak-gw06 kernel: CPU: 1 PID: 24179 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:02:28 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:02:28 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:02:28 oak-gw06 kernel: 00000000000080d0 0000000047a5dab5 ffff88006adf7858 ffffffff8168662f Aug 24 06:02:28 oak-gw06 kernel: ffff88006adf78e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:02:28 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88006adf78b8 0000000047a5dab5 Aug 24 06:02:28 oak-gw06 kernel: Call Trace: Aug 24 06:02:28 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:02:28 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:02:28 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:02:28 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:02:28 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 06:02:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 06:02:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:02:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:02:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:02:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:02:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:02:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:02:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:02:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:02:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:02:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:02:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:02:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:02:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:02:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:02:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:02:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:02:28 oak-gw06 kernel: Mem-Info: Aug 24 06:02:28 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1840144 inactive_file:147269 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72554 slab_unreclaimable:1234641#012 mapped:13291 shmem:49051 pagetables:1445 bounce:0#012 free:610631 free_pcp:62 free_cma:0 Aug 24 06:02:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:02:28 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:02:28 oak-gw06 kernel: Node 0 DMA32 free:1355800kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:644628kB inactive_file:192104kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9268kB shmem:31220kB slab_reclaimable:24972kB slab_unreclaimable:563356kB kernel_stack:1040kB pagetables:1140kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:02:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:02:28 oak-gw06 kernel: Node 0 Normal free:1070180kB min:323104kB low:403880kB high:484656kB active_anon:40452kB inactive_anon:184660kB active_file:6715948kB inactive_file:396972kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:43896kB shmem:164984kB slab_reclaimable:265244kB slab_unreclaimable:4375192kB kernel_stack:4688kB pagetables:4640kB unstable:0kB bounce:0kB free_pcp:308kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:02:28 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:02:28 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:02:28 oak-gw06 kernel: Node 0 DMA32: 11150*4kB (UEM) 11441*8kB (UEM) 6437*16kB (UEM) 12027*32kB (UEM) 7393*64kB (UEM) 1712*128kB (UEM) 158*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1357232kB Aug 24 06:02:28 oak-gw06 kernel: Node 0 Normal: 42482*4kB (UEM) 64383*8kB (UEM) 16251*16kB (UEM) 2793*32kB (UEM) 474*64kB (UE) 52*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1071376kB Aug 24 06:02:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:02:28 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:02:28 oak-gw06 kernel: 2036408 total pagecache pages Aug 24 06:02:28 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:02:28 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:02:28 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:02:28 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:02:28 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:02:28 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:02:28 oak-gw06 kernel: 127313 pages reserved Aug 24 06:02:28 oak-gw06 kernel: kworker/u16:2: page allocation failure: order:7, mode:0x80d0 Aug 24 06:02:28 oak-gw06 kernel: CPU: 1 PID: 24179 Comm: kworker/u16:2 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:02:28 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:02:28 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:02:28 oak-gw06 kernel: 00000000000080d0 0000000047a5dab5 ffff88006adf7808 ffffffff8168662f Aug 24 06:02:28 oak-gw06 kernel: ffff88006adf7898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 24 06:02:28 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88006adf7868 0000000047a5dab5 Aug 24 06:02:28 oak-gw06 kernel: Call Trace: Aug 24 06:02:28 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:02:28 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:02:28 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 24 06:02:28 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:02:28 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:02:28 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 06:02:28 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 06:02:28 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 06:02:28 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 06:02:28 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:02:28 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:02:28 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:02:28 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:02:28 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:02:28 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:02:28 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:02:28 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:02:28 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:02:28 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:02:28 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:02:28 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:02:28 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:02:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:02:28 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:02:28 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:02:28 oak-gw06 kernel: Mem-Info: Aug 24 06:02:28 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1840079 inactive_file:147269 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72554 slab_unreclaimable:1234641#012 mapped:13291 shmem:49051 pagetables:1445 bounce:0#012 free:610663 free_pcp:159 free_cma:0 Aug 24 06:02:28 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:02:29 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:02:29 oak-gw06 kernel: Node 0 DMA32 free:1355800kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:644628kB inactive_file:192104kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9268kB shmem:31220kB slab_reclaimable:24972kB slab_unreclaimable:563356kB kernel_stack:1040kB pagetables:1140kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:02:29 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:02:29 oak-gw06 kernel: Node 0 Normal free:1070960kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6715688kB inactive_file:396972kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:43896kB shmem:164984kB slab_reclaimable:265244kB slab_unreclaimable:4375192kB kernel_stack:4688kB pagetables:4640kB unstable:0kB bounce:0kB free_pcp:752kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:02:29 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:02:29 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:02:29 oak-gw06 kernel: Node 0 DMA32: 11150*4kB (UEM) 11441*8kB (UEM) 6437*16kB (UEM) 12027*32kB (UEM) 7393*64kB (UEM) 1712*128kB (UEM) 158*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1357232kB Aug 24 06:02:29 oak-gw06 kernel: Node 0 Normal: 42482*4kB (UEM) 64383*8kB (UEM) 16251*16kB (UEM) 2793*32kB (UEM) 474*64kB (UE) 52*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1071376kB Aug 24 06:02:29 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:02:29 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:02:29 oak-gw06 kernel: 2036311 total pagecache pages Aug 24 06:02:29 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:02:29 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:02:29 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:02:29 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:02:29 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:02:29 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:02:29 oak-gw06 kernel: 127313 pages reserved Aug 24 06:07:29 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:07:29 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:07:29 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:07:29 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:07:29 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97858 ffffffff8168662f Aug 24 06:07:29 oak-gw06 kernel: ffff880172a978e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:07:29 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a978b8 00000000f1450c91 Aug 24 06:07:29 oak-gw06 kernel: Call Trace: Aug 24 06:07:29 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:07:29 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:07:29 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:07:29 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:07:29 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 06:07:29 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 06:07:29 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:07:29 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:07:29 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:07:29 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:07:29 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:07:29 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:07:29 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:07:29 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:07:29 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:07:29 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:07:29 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:07:29 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:07:29 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:07:29 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:07:29 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:07:29 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:07:29 oak-gw06 kernel: Mem-Info: Aug 24 06:07:29 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1840016 inactive_file:147272 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72554 slab_unreclaimable:1234703#012 mapped:13299 shmem:49051 pagetables:1445 bounce:0#012 free:611156 free_pcp:62 free_cma:0 Aug 24 06:07:29 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:07:29 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:07:29 oak-gw06 kernel: Node 0 DMA32 free:1357344kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:644628kB inactive_file:192096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9264kB shmem:31220kB slab_reclaimable:24972kB slab_unreclaimable:563340kB kernel_stack:1040kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:07:29 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:07:29 oak-gw06 kernel: Node 0 Normal free:1070752kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6715436kB inactive_file:396992kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:43932kB shmem:164984kB slab_reclaimable:265244kB slab_unreclaimable:4375456kB kernel_stack:4640kB pagetables:4644kB unstable:0kB bounce:0kB free_pcp:744kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:07:29 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:07:29 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:07:29 oak-gw06 kernel: Node 0 DMA32: 11151*4kB (UEM) 11441*8kB (UEM) 6444*16kB (UEM) 12027*32kB (UEM) 7393*64kB (UEM) 1712*128kB (UEM) 158*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1357348kB Aug 24 06:07:29 oak-gw06 kernel: Node 0 Normal: 42482*4kB (UEM) 64439*8kB (UEM) 16229*16kB (UEM) 2795*32kB (UEM) 474*64kB (UE) 52*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1071536kB Aug 24 06:07:29 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:07:29 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:07:29 oak-gw06 kernel: 2036345 total pagecache pages Aug 24 06:07:29 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:07:29 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:07:29 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:07:29 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:07:29 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:07:29 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:07:29 oak-gw06 kernel: 127313 pages reserved Aug 24 06:07:29 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:07:29 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:07:29 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:07:29 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:07:29 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97808 ffffffff8168662f Aug 24 06:07:29 oak-gw06 kernel: ffff880172a97898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:07:29 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a97868 00000000f1450c91 Aug 24 06:07:29 oak-gw06 kernel: Call Trace: Aug 24 06:07:29 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:07:29 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:07:29 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:07:29 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:07:29 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 06:07:29 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 06:07:29 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 06:07:29 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 06:07:29 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:07:29 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:07:29 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:07:29 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:07:29 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:07:29 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:07:29 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:07:29 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:07:29 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:07:29 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:07:29 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:07:29 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:07:29 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:07:29 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:07:29 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:07:29 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:07:29 oak-gw06 kernel: Mem-Info: Aug 24 06:07:29 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1839951 inactive_file:147272 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72554 slab_unreclaimable:1234703#012 mapped:13299 shmem:49051 pagetables:1445 bounce:0#012 free:611322 free_pcp:92 free_cma:0 Aug 24 06:07:29 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:07:29 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:07:29 oak-gw06 kernel: Node 0 DMA32 free:1357344kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:644628kB inactive_file:192096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9264kB shmem:31220kB slab_reclaimable:24972kB slab_unreclaimable:563340kB kernel_stack:1040kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:07:29 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:07:29 oak-gw06 kernel: Node 0 Normal free:1072052kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6715176kB inactive_file:396992kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:43932kB shmem:164984kB slab_reclaimable:265244kB slab_unreclaimable:4375456kB kernel_stack:4640kB pagetables:4644kB unstable:0kB bounce:0kB free_pcp:360kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:07:29 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:07:29 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:07:29 oak-gw06 kernel: Node 0 DMA32: 11151*4kB (UEM) 11441*8kB (UEM) 6447*16kB (UEM) 12027*32kB (UEM) 7393*64kB (UEM) 1712*128kB (UEM) 158*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1357396kB Aug 24 06:07:30 oak-gw06 kernel: Node 0 Normal: 42577*4kB (UEM) 64440*8kB (UEM) 16229*16kB (UEM) 2795*32kB (UEM) 474*64kB (UE) 52*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1071924kB Aug 24 06:07:30 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:07:30 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:07:30 oak-gw06 kernel: 2036248 total pagecache pages Aug 24 06:07:30 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:07:30 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:07:30 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:07:30 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:07:30 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:07:30 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:07:30 oak-gw06 kernel: 127313 pages reserved Aug 24 06:12:29 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:12:29 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:12:29 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:12:29 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:12:29 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97858 ffffffff8168662f Aug 24 06:12:29 oak-gw06 kernel: ffff880172a978e8 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Aug 24 06:12:29 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 00000000f1450c91 Aug 24 06:12:29 oak-gw06 kernel: Call Trace: Aug 24 06:12:29 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:12:29 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:12:29 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Aug 24 06:12:29 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:12:29 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:12:29 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 06:12:29 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 06:12:29 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:12:29 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:12:29 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:12:29 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:12:29 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:12:29 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:12:29 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:12:29 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:12:29 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:12:29 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:12:29 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:12:29 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:12:29 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:12:29 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:12:29 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:12:29 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:12:29 oak-gw06 kernel: Mem-Info: Aug 24 06:12:29 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1839845 inactive_file:147260 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72554 slab_unreclaimable:1234665#012 mapped:13315 shmem:49051 pagetables:1446 bounce:0#012 free:611333 free_pcp:31 free_cma:0 Aug 24 06:12:29 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:12:29 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:12:29 oak-gw06 kernel: Node 0 DMA32 free:1357444kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:644628kB inactive_file:192096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9272kB shmem:31220kB slab_reclaimable:24972kB slab_unreclaimable:563292kB kernel_stack:1040kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:12:29 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:12:29 oak-gw06 kernel: Node 0 Normal free:1071280kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6714752kB inactive_file:396944kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:43988kB shmem:164984kB slab_reclaimable:265244kB slab_unreclaimable:4375352kB kernel_stack:4640kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:744kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:12:29 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:12:29 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:12:29 oak-gw06 kernel: Node 0 DMA32: 11167*4kB (UEM) 11448*8kB (UEM) 6452*16kB (UEM) 12027*32kB (UEM) 7393*64kB (UEM) 1712*128kB (UEM) 158*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1357596kB Aug 24 06:12:29 oak-gw06 kernel: Node 0 Normal: 42618*4kB (UEM) 64471*8kB (UEM) 16235*16kB (UEM) 2796*32kB (UEM) 474*64kB (UE) 52*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1072464kB Aug 24 06:12:29 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:12:29 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:12:29 oak-gw06 kernel: 2036130 total pagecache pages Aug 24 06:12:29 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:12:29 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:12:29 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:12:29 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:12:29 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:12:29 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:12:29 oak-gw06 kernel: 127313 pages reserved Aug 24 06:12:29 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:12:29 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:12:29 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:12:29 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:12:29 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97808 ffffffff8168662f Aug 24 06:12:29 oak-gw06 kernel: ffff880172a97898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:12:29 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a97868 00000000f1450c91 Aug 24 06:12:29 oak-gw06 kernel: Call Trace: Aug 24 06:12:29 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:12:29 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:12:29 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:12:29 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:12:29 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 06:12:29 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 06:12:29 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 06:12:29 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 06:12:29 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:12:29 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:12:29 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:12:29 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:12:29 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:12:29 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:12:29 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:12:29 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:12:29 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:12:29 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:12:29 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:12:29 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:12:29 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:12:29 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:12:29 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:12:29 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:12:29 oak-gw06 kernel: Mem-Info: Aug 24 06:12:29 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1839715 inactive_file:147260 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72554 slab_unreclaimable:1234665#012 mapped:13315 shmem:49051 pagetables:1446 bounce:0#012 free:611479 free_pcp:31 free_cma:0 Aug 24 06:12:29 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:12:29 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:12:29 oak-gw06 kernel: Node 0 DMA32 free:1357444kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:644628kB inactive_file:192096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9272kB shmem:31220kB slab_reclaimable:24972kB slab_unreclaimable:563292kB kernel_stack:1040kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:12:30 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:12:30 oak-gw06 kernel: Node 0 Normal free:1072580kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6714232kB inactive_file:396944kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:43988kB shmem:164984kB slab_reclaimable:265244kB slab_unreclaimable:4375352kB kernel_stack:4640kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:12:30 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:12:30 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:12:30 oak-gw06 kernel: Node 0 DMA32: 11167*4kB (UEM) 11448*8kB (UEM) 6452*16kB (UEM) 12027*32kB (UEM) 7393*64kB (UEM) 1712*128kB (UEM) 158*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1357596kB Aug 24 06:12:30 oak-gw06 kernel: Node 0 Normal: 42620*4kB (UEM) 64472*8kB (UEM) 16264*16kB (UEM) 2797*32kB (UEM) 474*64kB (UE) 52*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1072976kB Aug 24 06:12:30 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:12:30 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:12:30 oak-gw06 kernel: 2036033 total pagecache pages Aug 24 06:12:30 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:12:30 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:12:30 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:12:30 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:12:30 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:12:30 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:12:30 oak-gw06 kernel: 127313 pages reserved Aug 24 06:17:31 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:17:31 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:17:31 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:17:31 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:17:31 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97858 ffffffff8168662f Aug 24 06:17:31 oak-gw06 kernel: ffff880172a978e8 ffffffff81186ba0 0000000000000000 ffff88043ffd9000 Aug 24 06:17:31 oak-gw06 kernel: 0000000000000007 00000000000080d0 ffff880172a978e8 00000000f1450c91 Aug 24 06:17:31 oak-gw06 kernel: Call Trace: Aug 24 06:17:31 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:17:31 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:17:31 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:17:31 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:17:31 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 06:17:31 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 06:17:31 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:17:31 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:17:31 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:17:31 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:17:31 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:17:31 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:17:31 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:17:31 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:17:31 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:17:31 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:17:31 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:17:31 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:17:31 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:17:31 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:17:31 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:17:31 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:17:31 oak-gw06 kernel: Mem-Info: Aug 24 06:17:31 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1839526 inactive_file:147263 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72424 slab_unreclaimable:1234611#012 mapped:13326 shmem:49051 pagetables:1446 bounce:0#012 free:611959 free_pcp:159 free_cma:0 Aug 24 06:17:31 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:17:31 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:17:31 oak-gw06 kernel: Node 0 DMA32 free:1357676kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:644508kB inactive_file:192096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9268kB shmem:31220kB slab_reclaimable:24972kB slab_unreclaimable:563180kB kernel_stack:1040kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:17:31 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:17:31 oak-gw06 kernel: Node 0 Normal free:1073524kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6713596kB inactive_file:396956kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44036kB shmem:164984kB slab_reclaimable:264724kB slab_unreclaimable:4375248kB kernel_stack:4640kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:1380kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:17:31 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:17:31 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:17:31 oak-gw06 kernel: Node 0 DMA32: 14767*4kB (UEM) 13168*8kB (UEM) 6870*16kB (UEM) 11592*32kB (UEM) 7156*64kB (UEM) 1676*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1357724kB Aug 24 06:17:31 oak-gw06 kernel: Node 0 Normal: 42559*4kB (UEM) 64472*8kB (UEM) 16286*16kB (UEM) 2809*32kB (UEM) 474*64kB (UE) 52*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1073468kB Aug 24 06:17:31 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:17:31 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:17:31 oak-gw06 kernel: 2035848 total pagecache pages Aug 24 06:17:31 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:17:31 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:17:31 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:17:31 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:17:31 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:17:31 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:17:31 oak-gw06 kernel: 127313 pages reserved Aug 24 06:17:31 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:17:31 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:17:31 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:17:31 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:17:31 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97808 ffffffff8168662f Aug 24 06:17:31 oak-gw06 kernel: ffff880172a97898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:17:31 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a97868 00000000f1450c91 Aug 24 06:17:31 oak-gw06 kernel: Call Trace: Aug 24 06:17:31 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:17:31 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:17:31 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:17:31 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:17:31 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 06:17:31 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 06:17:31 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 06:17:31 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 06:17:31 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:17:31 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:17:31 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:17:31 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:17:31 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:17:31 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:17:31 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:17:31 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:17:31 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:17:31 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:17:31 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:17:31 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:17:31 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:17:31 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:17:31 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:17:31 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:17:31 oak-gw06 kernel: Mem-Info: Aug 24 06:17:31 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1839461 inactive_file:147263 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72424 slab_unreclaimable:1234611#012 mapped:13326 shmem:49051 pagetables:1446 bounce:0#012 free:612098 free_pcp:31 free_cma:0 Aug 24 06:17:31 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:17:31 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:17:31 oak-gw06 kernel: Node 0 DMA32 free:1357676kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:644508kB inactive_file:192096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9268kB shmem:31220kB slab_reclaimable:24972kB slab_unreclaimable:563180kB kernel_stack:1040kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:17:31 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:17:31 oak-gw06 kernel: Node 0 Normal free:1074824kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6713336kB inactive_file:396956kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44036kB shmem:164984kB slab_reclaimable:264724kB slab_unreclaimable:4375248kB kernel_stack:4640kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:17:31 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:17:31 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:17:31 oak-gw06 kernel: Node 0 DMA32: 14767*4kB (UEM) 13168*8kB (UEM) 6870*16kB (UEM) 11592*32kB (UEM) 7156*64kB (UEM) 1676*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1357724kB Aug 24 06:17:31 oak-gw06 kernel: Node 0 Normal: 42714*4kB (UEM) 64484*8kB (UEM) 16313*16kB (UEM) 2809*32kB (UEM) 474*64kB (UE) 52*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1074616kB Aug 24 06:17:31 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:17:31 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:17:31 oak-gw06 kernel: 2035751 total pagecache pages Aug 24 06:17:31 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:17:31 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:17:31 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:17:31 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:17:31 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:17:31 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:17:31 oak-gw06 kernel: 127313 pages reserved Aug 24 06:22:30 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 24 06:22:30 oak-gw06 kernel: CPU: 1 PID: 24242 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:22:30 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:22:30 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:22:30 oak-gw06 kernel: 00000000000080d0 000000005887b6d3 ffff88010378b858 ffffffff8168662f Aug 24 06:22:30 oak-gw06 kernel: ffff88010378b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:22:30 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88010378b8b8 000000005887b6d3 Aug 24 06:22:30 oak-gw06 kernel: Call Trace: Aug 24 06:22:30 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:22:30 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:22:30 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:22:30 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:22:30 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 06:22:30 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 06:22:30 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:22:30 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:22:30 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:22:30 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:22:30 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:22:30 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:22:30 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:22:30 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:22:30 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:22:30 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:22:30 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:22:30 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:22:30 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:22:30 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:22:30 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:22:30 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:22:30 oak-gw06 kernel: Mem-Info: Aug 24 06:22:30 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1839335 inactive_file:147271 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72399 slab_unreclaimable:1234595#012 mapped:13341 shmem:49051 pagetables:1446 bounce:0#012 free:612107 free_pcp:31 free_cma:0 Aug 24 06:22:30 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:22:30 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:22:30 oak-gw06 kernel: Node 0 DMA32 free:1357724kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:644508kB inactive_file:192096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9272kB shmem:31220kB slab_reclaimable:24916kB slab_unreclaimable:563180kB kernel_stack:1040kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:22:30 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:22:30 oak-gw06 kernel: Node 0 Normal free:1074104kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6712832kB inactive_file:396988kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44092kB shmem:164984kB slab_reclaimable:264680kB slab_unreclaimable:4375184kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:744kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:22:30 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:22:30 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:22:30 oak-gw06 kernel: Node 0 DMA32: 14767*4kB (UEM) 13167*8kB (UEM) 6871*16kB (UEM) 11592*32kB (UEM) 7156*64kB (UEM) 1676*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1357732kB Aug 24 06:22:30 oak-gw06 kernel: Node 0 Normal: 42668*4kB (UEM) 64547*8kB (UEM) 16321*16kB (UEM) 2809*32kB (UEM) 476*64kB (UE) 52*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1075192kB Aug 24 06:22:30 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:22:30 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:22:30 oak-gw06 kernel: 2035631 total pagecache pages Aug 24 06:22:30 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:22:30 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:22:30 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:22:30 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:22:30 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:22:30 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:22:30 oak-gw06 kernel: 127313 pages reserved Aug 24 06:22:30 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 24 06:22:30 oak-gw06 kernel: CPU: 1 PID: 24242 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:22:30 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:22:30 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:22:30 oak-gw06 kernel: 00000000000080d0 000000005887b6d3 ffff88010378b808 ffffffff8168662f Aug 24 06:22:30 oak-gw06 kernel: ffff88010378b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:22:30 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88010378b868 000000005887b6d3 Aug 24 06:22:30 oak-gw06 kernel: Call Trace: Aug 24 06:22:30 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:22:30 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:22:30 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:22:30 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:22:30 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 06:22:30 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 06:22:30 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 06:22:30 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 06:22:30 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:22:30 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:22:30 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:22:30 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:22:30 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:22:30 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:22:30 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:22:30 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:22:30 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:22:30 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:22:30 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:22:30 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:22:30 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:22:30 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:22:30 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:22:30 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:22:30 oak-gw06 kernel: Mem-Info: Aug 24 06:22:30 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1839205 inactive_file:147271 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72399 slab_unreclaimable:1234595#012 mapped:13341 shmem:49051 pagetables:1446 bounce:0#012 free:612255 free_pcp:31 free_cma:0 Aug 24 06:22:30 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:22:30 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:22:30 oak-gw06 kernel: Node 0 DMA32 free:1357724kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:644508kB inactive_file:192096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9272kB shmem:31220kB slab_reclaimable:24916kB slab_unreclaimable:563180kB kernel_stack:1040kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:22:30 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:22:30 oak-gw06 kernel: Node 0 Normal free:1075404kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6712312kB inactive_file:396988kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44092kB shmem:164984kB slab_reclaimable:264680kB slab_unreclaimable:4375184kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:22:30 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:22:31 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:22:31 oak-gw06 kernel: Node 0 DMA32: 14767*4kB (UEM) 13167*8kB (UEM) 6871*16kB (UEM) 11592*32kB (UEM) 7156*64kB (UEM) 1676*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1357732kB Aug 24 06:22:31 oak-gw06 kernel: Node 0 Normal: 42668*4kB (UEM) 64609*8kB (UEM) 16320*16kB (UEM) 2810*32kB (UEM) 476*64kB (UE) 52*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1075704kB Aug 24 06:22:31 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:22:31 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:22:31 oak-gw06 kernel: 2035534 total pagecache pages Aug 24 06:22:31 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:22:31 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:22:31 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:22:31 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:22:31 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:22:31 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:22:31 oak-gw06 kernel: 127313 pages reserved Aug 24 06:27:31 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:27:31 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:27:31 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:27:31 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:27:31 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97858 ffffffff8168662f Aug 24 06:27:31 oak-gw06 kernel: ffff880172a978e8 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Aug 24 06:27:31 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 00000000f1450c91 Aug 24 06:27:31 oak-gw06 kernel: Call Trace: Aug 24 06:27:31 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:27:31 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:27:31 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Aug 24 06:27:31 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:27:31 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:27:31 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 06:27:31 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 06:27:31 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:27:31 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:27:31 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:27:31 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:27:31 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:27:31 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:27:31 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:27:31 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:27:31 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:27:31 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:27:31 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:27:31 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:27:31 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:27:31 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:27:31 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:27:31 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:27:31 oak-gw06 kernel: Mem-Info: Aug 24 06:27:31 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1839079 inactive_file:147276 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72399 slab_unreclaimable:1234603#012 mapped:13349 shmem:49051 pagetables:1446 bounce:0#012 free:612376 free_pcp:62 free_cma:0 Aug 24 06:27:31 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:27:31 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:27:31 oak-gw06 kernel: Node 0 DMA32 free:1357876kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:644508kB inactive_file:192096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9268kB shmem:31220kB slab_reclaimable:24916kB slab_unreclaimable:563164kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:27:31 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:27:31 oak-gw06 kernel: Node 0 Normal free:1075092kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6711808kB inactive_file:397008kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44128kB shmem:164984kB slab_reclaimable:264680kB slab_unreclaimable:4375232kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:868kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:27:31 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:27:31 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:27:31 oak-gw06 kernel: Node 0 DMA32: 14767*4kB (UEM) 13167*8kB (UEM) 6880*16kB (UEM) 11592*32kB (UEM) 7156*64kB (UEM) 1676*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1357876kB Aug 24 06:27:31 oak-gw06 kernel: Node 0 Normal: 42632*4kB (UEM) 64609*8kB (UEM) 16344*16kB (UEM) 2810*32kB (UEM) 476*64kB (UE) 52*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1075944kB Aug 24 06:27:31 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:27:31 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:27:31 oak-gw06 kernel: 2035380 total pagecache pages Aug 24 06:27:31 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:27:31 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:27:31 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:27:31 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:27:31 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:27:31 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:27:31 oak-gw06 kernel: 127313 pages reserved Aug 24 06:27:31 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:27:31 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:27:31 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:27:31 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:27:31 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97808 ffffffff8168662f Aug 24 06:27:31 oak-gw06 kernel: ffff880172a97898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:27:31 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a97868 00000000f1450c91 Aug 24 06:27:31 oak-gw06 kernel: Call Trace: Aug 24 06:27:31 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:27:31 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:27:31 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:27:31 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:27:31 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 06:27:31 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 06:27:31 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 06:27:31 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 06:27:31 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:27:31 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:27:31 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:27:31 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:27:31 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:27:31 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:27:31 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:27:31 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:27:31 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:27:31 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:27:31 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:27:31 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:27:31 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:27:31 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:27:31 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:27:31 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:27:31 oak-gw06 kernel: Mem-Info: Aug 24 06:27:31 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1838884 inactive_file:147276 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72399 slab_unreclaimable:1234537#012 mapped:13349 shmem:49051 pagetables:1446 bounce:0#012 free:612540 free_pcp:197 free_cma:0 Aug 24 06:27:31 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:27:31 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:27:31 oak-gw06 kernel: Node 0 DMA32 free:1357876kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:644508kB inactive_file:192096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9268kB shmem:31220kB slab_reclaimable:24916kB slab_unreclaimable:563164kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:27:31 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:27:31 oak-gw06 kernel: Node 0 Normal free:1075748kB min:323104kB low:403880kB high:484656kB active_anon:40452kB inactive_anon:184660kB active_file:6711028kB inactive_file:397008kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44128kB shmem:164984kB slab_reclaimable:264680kB slab_unreclaimable:4374968kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:936kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:27:32 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:27:32 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:27:32 oak-gw06 kernel: Node 0 DMA32: 14767*4kB (UEM) 13167*8kB (UEM) 6880*16kB (UEM) 11592*32kB (UEM) 7156*64kB (UEM) 1676*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1357876kB Aug 24 06:27:32 oak-gw06 kernel: Node 0 Normal: 42669*4kB (UEM) 64646*8kB (UEM) 16372*16kB (UEM) 2810*32kB (UEM) 476*64kB (UE) 52*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1076836kB Aug 24 06:27:32 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:27:32 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:27:32 oak-gw06 kernel: 2035186 total pagecache pages Aug 24 06:27:32 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:27:32 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:27:32 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:27:32 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:27:32 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:27:32 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:27:32 oak-gw06 kernel: 127313 pages reserved Aug 24 06:32:32 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:32:32 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:32:32 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:32:32 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:32:32 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97858 ffffffff8168662f Aug 24 06:32:32 oak-gw06 kernel: ffff880172a978e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:32:32 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a978b8 00000000f1450c91 Aug 24 06:32:32 oak-gw06 kernel: Call Trace: Aug 24 06:32:32 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:32:32 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:32:32 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:32:32 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:32:32 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 06:32:32 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 06:32:32 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:32:32 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:32:32 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:32:32 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:32:32 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:32:32 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:32:32 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:32:32 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:32:32 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:32:32 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:32:32 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:32:32 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:32:32 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:32:32 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:32:32 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:32:32 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:32:32 oak-gw06 kernel: Mem-Info: Aug 24 06:32:32 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1838635 inactive_file:147284 isolated_file:0#012 unevictable:0 dirty:0 writeback:1 unstable:0#012 slab_reclaimable:72389 slab_unreclaimable:1234310#012 mapped:13361 shmem:49051 pagetables:1446 bounce:0#012 free:613106 free_pcp:159 free_cma:0 Aug 24 06:32:32 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:32:32 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:32:32 oak-gw06 kernel: Node 0 DMA32 free:1358468kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:643996kB inactive_file:192096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9272kB shmem:31220kB slab_reclaimable:24908kB slab_unreclaimable:562628kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:512kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:32:32 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:32:32 oak-gw06 kernel: Node 0 Normal free:1077320kB min:323104kB low:403880kB high:484656kB active_anon:40192kB inactive_anon:184660kB active_file:6710544kB inactive_file:397040kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:4kB mapped:44172kB shmem:164984kB slab_reclaimable:264648kB slab_unreclaimable:4374596kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:236kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:32:32 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:32:32 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:32:32 oak-gw06 kernel: Node 0 DMA32: 14765*4kB (UEM) 13170*8kB (UEM) 6884*16kB (UEM) 11604*32kB (UEM) 7156*64kB (UEM) 1677*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1358468kB Aug 24 06:32:32 oak-gw06 kernel: Node 0 Normal: 42807*4kB (UEM) 64659*8kB (UEM) 16375*16kB (UEM) 2810*32kB (UEM) 475*64kB (UE) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1077604kB Aug 24 06:32:32 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:32:32 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:32:32 oak-gw06 kernel: 2034976 total pagecache pages Aug 24 06:32:32 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:32:32 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:32:32 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:32:32 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:32:32 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:32:32 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:32:32 oak-gw06 kernel: 127313 pages reserved Aug 24 06:32:32 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:32:32 oak-gw06 kernel: CPU: 2 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:32:32 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:32:32 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:32:32 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97808 ffffffff8168662f Aug 24 06:32:32 oak-gw06 kernel: ffff880172a97898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 24 06:32:32 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a97868 00000000f1450c91 Aug 24 06:32:32 oak-gw06 kernel: Call Trace: Aug 24 06:32:32 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:32:32 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:32:32 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 24 06:32:32 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:32:32 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:32:32 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 06:32:32 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 06:32:32 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 06:32:32 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 06:32:32 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:32:32 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:32:32 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:32:32 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:32:32 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:32:32 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:32:32 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:32:32 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:32:32 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:32:32 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:32:32 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:32:32 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:32:32 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:32:32 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:32:32 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:32:32 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:32:32 oak-gw06 kernel: Mem-Info: Aug 24 06:32:32 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1838635 inactive_file:147284 isolated_file:0#012 unevictable:0 dirty:0 writeback:1 unstable:0#012 slab_reclaimable:72389 slab_unreclaimable:1234310#012 mapped:13361 shmem:49051 pagetables:1446 bounce:0#012 free:613216 free_pcp:31 free_cma:0 Aug 24 06:32:32 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:32:32 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:32:32 oak-gw06 kernel: Node 0 DMA32 free:1358972kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:643996kB inactive_file:192096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9272kB shmem:31220kB slab_reclaimable:24908kB slab_unreclaimable:562628kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:32:32 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:32:33 oak-gw06 kernel: Node 0 Normal free:1077256kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6710544kB inactive_file:397040kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:4kB mapped:44172kB shmem:164984kB slab_reclaimable:264648kB slab_unreclaimable:4374596kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:300kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:32:33 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:32:33 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:32:33 oak-gw06 kernel: Node 0 DMA32: 14763*4kB (UEM) 13177*8kB (UEM) 6913*16kB (UEM) 11604*32kB (UEM) 7156*64kB (UEM) 1677*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1358980kB Aug 24 06:32:33 oak-gw06 kernel: Node 0 Normal: 42807*4kB (UEM) 64659*8kB (UEM) 16375*16kB (UEM) 2810*32kB (UEM) 475*64kB (UE) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1077604kB Aug 24 06:32:33 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:32:33 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:32:33 oak-gw06 kernel: 2034976 total pagecache pages Aug 24 06:32:33 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:32:33 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:32:33 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:32:33 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:32:33 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:32:33 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:32:33 oak-gw06 kernel: 127313 pages reserved Aug 24 06:37:33 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:37:33 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:37:33 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:37:33 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:37:33 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97858 ffffffff8168662f Aug 24 06:37:33 oak-gw06 kernel: ffff880172a978e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:37:33 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a978b8 00000000f1450c91 Aug 24 06:37:33 oak-gw06 kernel: Call Trace: Aug 24 06:37:33 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:37:33 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:37:33 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:37:33 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:37:33 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 06:37:33 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 06:37:33 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:37:33 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:37:33 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:37:33 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:37:33 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:37:33 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:37:33 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:37:33 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:37:33 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:37:33 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:37:33 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:37:33 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:37:33 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:37:33 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:37:33 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:37:33 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:37:33 oak-gw06 kernel: Mem-Info: Aug 24 06:37:33 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1838570 inactive_file:147289 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72389 slab_unreclaimable:1234330#012 mapped:13373 shmem:49051 pagetables:1446 bounce:0#012 free:613428 free_pcp:31 free_cma:0 Aug 24 06:37:33 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:37:33 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:37:33 oak-gw06 kernel: Node 0 DMA32 free:1358996kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:643996kB inactive_file:192096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9272kB shmem:31220kB slab_reclaimable:24908kB slab_unreclaimable:562612kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:37:33 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:37:33 oak-gw06 kernel: Node 0 Normal free:1078080kB min:323104kB low:403880kB high:484656kB active_anon:40452kB inactive_anon:184660kB active_file:6710284kB inactive_file:397060kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44220kB shmem:164984kB slab_reclaimable:264648kB slab_unreclaimable:4374692kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:236kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:37:33 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:37:33 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:37:33 oak-gw06 kernel: Node 0 DMA32: 14763*4kB (UEM) 13177*8kB (UEM) 6914*16kB (UEM) 11604*32kB (UEM) 7156*64kB (UEM) 1677*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1358996kB Aug 24 06:37:33 oak-gw06 kernel: Node 0 Normal: 42807*4kB (UEM) 64707*8kB (UEM) 16383*16kB (UEM) 2810*32kB (UEM) 475*64kB (UE) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1078116kB Aug 24 06:37:33 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:37:33 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:37:33 oak-gw06 kernel: 2034884 total pagecache pages Aug 24 06:37:33 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:37:33 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:37:33 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:37:33 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:37:33 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:37:33 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:37:33 oak-gw06 kernel: 127313 pages reserved Aug 24 06:37:33 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:37:33 oak-gw06 kernel: CPU: 7 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:37:33 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:37:33 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:37:33 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97808 ffffffff8168662f Aug 24 06:37:33 oak-gw06 kernel: ffff880172a97898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:37:33 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a97868 00000000f1450c91 Aug 24 06:37:33 oak-gw06 kernel: Call Trace: Aug 24 06:37:33 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:37:33 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:37:33 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:37:33 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:37:33 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 06:37:33 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 06:37:33 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 06:37:33 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 06:37:33 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:37:33 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:37:33 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:37:33 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:37:33 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:37:33 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:37:33 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:37:33 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:37:33 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:37:33 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:37:33 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:37:33 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:37:33 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:37:33 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:37:33 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:37:33 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:37:33 oak-gw06 kernel: Mem-Info: Aug 24 06:37:33 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1838442 inactive_file:147289 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72389 slab_unreclaimable:1234302#012 mapped:13373 shmem:49051 pagetables:1446 bounce:0#012 free:613511 free_pcp:31 free_cma:0 Aug 24 06:37:33 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:37:33 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:37:33 oak-gw06 kernel: Node 0 DMA32 free:1358996kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:643996kB inactive_file:192096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9272kB shmem:31220kB slab_reclaimable:24908kB slab_unreclaimable:562612kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:37:33 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:37:34 oak-gw06 kernel: Node 0 Normal free:1079156kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6709772kB inactive_file:397060kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44220kB shmem:164984kB slab_reclaimable:264648kB slab_unreclaimable:4374580kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:37:34 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:37:34 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:37:34 oak-gw06 kernel: Node 0 DMA32: 14763*4kB (UEM) 13177*8kB (UEM) 6914*16kB (UEM) 11604*32kB (UEM) 7156*64kB (UEM) 1677*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1358996kB Aug 24 06:37:34 oak-gw06 kernel: Node 0 Normal: 42807*4kB (UEM) 64713*8kB (UEM) 16413*16kB (UEM) 2810*32kB (UEM) 475*64kB (UE) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1078644kB Aug 24 06:37:34 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:37:34 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:37:34 oak-gw06 kernel: 2034756 total pagecache pages Aug 24 06:37:34 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:37:34 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:37:34 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:37:34 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:37:34 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:37:34 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:37:34 oak-gw06 kernel: 127313 pages reserved Aug 24 06:42:34 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:42:34 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:42:34 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:42:34 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:42:34 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97858 ffffffff8168662f Aug 24 06:42:34 oak-gw06 kernel: ffff880172a978e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:42:34 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a978b8 00000000f1450c91 Aug 24 06:42:34 oak-gw06 kernel: Call Trace: Aug 24 06:42:34 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:42:34 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:42:34 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:42:34 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:42:34 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 06:42:34 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 06:42:34 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:42:34 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:42:34 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:42:34 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:42:34 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:42:34 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:42:34 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:42:34 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:42:34 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:42:34 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:42:34 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:42:34 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:42:34 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:42:34 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:42:34 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:42:34 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:42:34 oak-gw06 kernel: Mem-Info: Aug 24 06:42:34 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1838380 inactive_file:147169 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72389 slab_unreclaimable:1234238#012 mapped:13385 shmem:49051 pagetables:1446 bounce:0#012 free:613471 free_pcp:31 free_cma:0 Aug 24 06:42:34 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:42:34 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:42:34 oak-gw06 kernel: Node 0 DMA32 free:1359516kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:643996kB inactive_file:191592kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9276kB shmem:31220kB slab_reclaimable:24908kB slab_unreclaimable:562612kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:42:34 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:42:34 oak-gw06 kernel: Node 0 Normal free:1077768kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6709524kB inactive_file:397084kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44264kB shmem:164984kB slab_reclaimable:264648kB slab_unreclaimable:4374324kB kernel_stack:4672kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:744kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:42:34 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:42:34 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:42:34 oak-gw06 kernel: Node 0 DMA32: 14763*4kB (UEM) 13177*8kB (UEM) 6916*16kB (UEM) 11618*32kB (UEM) 7155*64kB (UEM) 1678*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1359540kB Aug 24 06:42:34 oak-gw06 kernel: Node 0 Normal: 42795*4kB (UEM) 64712*8kB (UEM) 16427*16kB (UEM) 2810*32kB (UEM) 475*64kB (UE) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1078812kB Aug 24 06:42:34 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:42:34 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:42:34 oak-gw06 kernel: 2034546 total pagecache pages Aug 24 06:42:34 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:42:34 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:42:34 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:42:34 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:42:34 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:42:34 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:42:34 oak-gw06 kernel: 127313 pages reserved Aug 24 06:42:34 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:42:34 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:42:34 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:42:34 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:42:34 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97808 ffffffff8168662f Aug 24 06:42:34 oak-gw06 kernel: ffff880172a97898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:42:34 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a97868 00000000f1450c91 Aug 24 06:42:34 oak-gw06 kernel: Call Trace: Aug 24 06:42:34 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:42:34 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:42:34 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:42:34 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:42:34 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 06:42:34 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 06:42:34 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 06:42:34 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 06:42:34 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:42:34 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:42:34 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:42:34 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:42:34 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:42:34 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:42:34 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:42:34 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:42:34 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:42:34 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:42:34 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:42:34 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:42:34 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:42:34 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:42:34 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:42:34 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:42:34 oak-gw06 kernel: Mem-Info: Aug 24 06:42:34 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1838315 inactive_file:147169 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72389 slab_unreclaimable:1234238#012 mapped:13385 shmem:49051 pagetables:1446 bounce:0#012 free:613619 free_pcp:31 free_cma:0 Aug 24 06:42:34 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:42:34 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:42:34 oak-gw06 kernel: Node 0 DMA32 free:1359516kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:643996kB inactive_file:191592kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9276kB shmem:31220kB slab_reclaimable:24908kB slab_unreclaimable:562612kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:42:35 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:42:35 oak-gw06 kernel: Node 0 Normal free:1079068kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6709264kB inactive_file:397084kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44264kB shmem:164984kB slab_reclaimable:264648kB slab_unreclaimable:4374324kB kernel_stack:4672kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:240kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:42:35 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:42:35 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:42:35 oak-gw06 kernel: Node 0 DMA32: 14763*4kB (UEM) 13177*8kB (UEM) 6916*16kB (UEM) 11618*32kB (UEM) 7155*64kB (UEM) 1678*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1359540kB Aug 24 06:42:35 oak-gw06 kernel: Node 0 Normal: 42950*4kB (UEM) 64712*8kB (UEM) 16428*16kB (UEM) 2810*32kB (UEM) 475*64kB (UE) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1079448kB Aug 24 06:42:35 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:42:35 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:42:35 oak-gw06 kernel: 2034449 total pagecache pages Aug 24 06:42:35 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:42:35 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:42:35 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:42:35 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:42:35 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:42:35 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:42:35 oak-gw06 kernel: 127313 pages reserved Aug 24 06:47:34 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:47:34 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:47:34 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:47:34 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:47:34 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97858 ffffffff8168662f Aug 24 06:47:34 oak-gw06 kernel: ffff880172a978e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:47:34 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a978b8 00000000f1450c91 Aug 24 06:47:34 oak-gw06 kernel: Call Trace: Aug 24 06:47:34 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:47:34 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:47:34 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:47:34 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:47:34 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 06:47:34 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 06:47:34 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:47:34 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:47:34 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:47:34 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:47:34 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:47:34 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:47:34 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:47:34 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:47:34 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:47:34 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:47:34 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:47:34 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:47:34 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:47:34 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:47:34 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:47:34 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:47:34 oak-gw06 kernel: Mem-Info: Aug 24 06:47:34 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1838187 inactive_file:147172 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72389 slab_unreclaimable:1234314#012 mapped:13396 shmem:49051 pagetables:1446 bounce:0#012 free:613597 free_pcp:159 free_cma:0 Aug 24 06:47:34 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:47:34 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:47:34 oak-gw06 kernel: Node 0 DMA32 free:1359636kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:643996kB inactive_file:191584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9272kB shmem:31220kB slab_reclaimable:24908kB slab_unreclaimable:562596kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:47:34 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:47:34 oak-gw06 kernel: Node 0 Normal free:1078216kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6708752kB inactive_file:397104kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44312kB shmem:164984kB slab_reclaimable:264648kB slab_unreclaimable:4374644kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:1132kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:47:34 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:47:34 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:47:34 oak-gw06 kernel: Node 0 DMA32: 14763*4kB (UEM) 13177*8kB (UEM) 6922*16kB (UEM) 11618*32kB (UEM) 7155*64kB (UEM) 1678*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1359636kB Aug 24 06:47:34 oak-gw06 kernel: Node 0 Normal: 42945*4kB (UEM) 64712*8kB (UEM) 16403*16kB (UEM) 2810*32kB (UEM) 475*64kB (UE) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1079028kB Aug 24 06:47:34 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:47:34 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:47:34 oak-gw06 kernel: 2034386 total pagecache pages Aug 24 06:47:34 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:47:34 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:47:34 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:47:34 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:47:34 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:47:34 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:47:34 oak-gw06 kernel: 127313 pages reserved Aug 24 06:47:34 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:47:34 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:47:34 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:47:34 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:47:34 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97808 ffffffff8168662f Aug 24 06:47:34 oak-gw06 kernel: ffff880172a97898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:47:34 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a97868 00000000f1450c91 Aug 24 06:47:34 oak-gw06 kernel: Call Trace: Aug 24 06:47:34 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:47:34 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:47:34 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:47:34 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:47:34 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 06:47:34 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 06:47:34 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 06:47:34 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 06:47:34 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:47:34 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:47:34 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:47:34 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:47:34 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:47:34 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:47:34 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:47:34 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:47:34 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:47:34 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:47:34 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:47:34 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:47:34 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:47:34 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:47:34 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:47:34 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:47:34 oak-gw06 kernel: Mem-Info: Aug 24 06:47:34 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1838187 inactive_file:147107 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72389 slab_unreclaimable:1234314#012 mapped:13396 shmem:49051 pagetables:1446 bounce:0#012 free:613761 free_pcp:31 free_cma:0 Aug 24 06:47:34 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:47:34 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:47:34 oak-gw06 kernel: Node 0 DMA32 free:1359636kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:643996kB inactive_file:191584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9272kB shmem:31220kB slab_reclaimable:24908kB slab_unreclaimable:562596kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:47:35 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:47:35 oak-gw06 kernel: Node 0 Normal free:1079516kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6708752kB inactive_file:396844kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44312kB shmem:164984kB slab_reclaimable:264648kB slab_unreclaimable:4374644kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:47:35 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:47:35 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:47:35 oak-gw06 kernel: Node 0 DMA32: 14799*4kB (UEM) 13177*8kB (UEM) 6927*16kB (UEM) 11618*32kB (UEM) 7155*64kB (UEM) 1678*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1359860kB Aug 24 06:47:35 oak-gw06 kernel: Node 0 Normal: 42974*4kB (UEM) 64730*8kB (UEM) 16431*16kB (UEM) 2811*32kB (UEM) 475*64kB (UE) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1079768kB Aug 24 06:47:35 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:47:35 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:47:35 oak-gw06 kernel: 2034289 total pagecache pages Aug 24 06:47:35 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:47:35 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:47:35 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:47:35 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:47:35 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:47:35 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:47:35 oak-gw06 kernel: 127313 pages reserved Aug 24 06:52:35 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 24 06:52:35 oak-gw06 kernel: CPU: 1 PID: 24242 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:52:35 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:52:35 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:52:35 oak-gw06 kernel: 00000000000080d0 000000005887b6d3 ffff88010378b858 ffffffff8168662f Aug 24 06:52:35 oak-gw06 kernel: ffff88010378b8e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:52:35 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88010378b8b8 000000005887b6d3 Aug 24 06:52:35 oak-gw06 kernel: Call Trace: Aug 24 06:52:35 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:52:35 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:52:35 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:52:35 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:52:35 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 06:52:35 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 06:52:35 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:52:35 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:52:35 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:52:35 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:52:35 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:52:35 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:52:35 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:52:35 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:52:35 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:52:35 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:52:35 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:52:35 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:52:35 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:52:35 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:52:35 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:52:35 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:52:35 oak-gw06 kernel: Mem-Info: Aug 24 06:52:35 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1838016 inactive_file:147097 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72389 slab_unreclaimable:1234279#012 mapped:13409 shmem:49051 pagetables:1446 bounce:0#012 free:613917 free_pcp:31 free_cma:0 Aug 24 06:52:35 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:52:35 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:52:35 oak-gw06 kernel: Node 0 DMA32 free:1359860kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:643820kB inactive_file:191584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9276kB shmem:31220kB slab_reclaimable:24908kB slab_unreclaimable:562548kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:52:35 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:52:35 oak-gw06 kernel: Node 0 Normal free:1079568kB min:323104kB low:403880kB high:484656kB active_anon:40452kB inactive_anon:184660kB active_file:6708244kB inactive_file:396804kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44360kB shmem:164984kB slab_reclaimable:264648kB slab_unreclaimable:4374552kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:332kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:52:35 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:52:35 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:52:35 oak-gw06 kernel: Node 0 DMA32: 14799*4kB (UEM) 13177*8kB (UEM) 6927*16kB (UEM) 11618*32kB (UEM) 7155*64kB (UEM) 1678*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1359860kB Aug 24 06:52:35 oak-gw06 kernel: Node 0 Normal: 42979*4kB (UEM) 64740*8kB (UEM) 16457*16kB (UEM) 2812*32kB (UEM) 475*64kB (UE) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1080316kB Aug 24 06:52:35 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:52:35 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:52:35 oak-gw06 kernel: 2034140 total pagecache pages Aug 24 06:52:35 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:52:35 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:52:35 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:52:35 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:52:35 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:52:35 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:52:35 oak-gw06 kernel: 127313 pages reserved Aug 24 06:52:35 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 24 06:52:35 oak-gw06 kernel: CPU: 1 PID: 24242 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:52:35 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:52:35 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:52:35 oak-gw06 kernel: 00000000000080d0 000000005887b6d3 ffff88010378b808 ffffffff8168662f Aug 24 06:52:35 oak-gw06 kernel: ffff88010378b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:52:35 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88010378b868 000000005887b6d3 Aug 24 06:52:35 oak-gw06 kernel: Call Trace: Aug 24 06:52:35 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:52:35 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:52:35 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:52:35 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:52:35 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 06:52:35 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 06:52:35 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 06:52:35 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 06:52:35 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:52:35 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:52:35 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:52:35 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:52:35 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:52:35 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:52:35 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:52:35 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:52:35 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:52:35 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:52:35 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:52:35 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:52:35 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:52:35 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:52:35 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:52:35 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:52:35 oak-gw06 kernel: Mem-Info: Aug 24 06:52:35 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1837886 inactive_file:147097 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72389 slab_unreclaimable:1234279#012 mapped:13409 shmem:49051 pagetables:1446 bounce:0#012 free:614062 free_pcp:31 free_cma:0 Aug 24 06:52:35 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:52:36 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:52:36 oak-gw06 kernel: Node 0 DMA32 free:1359860kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:643820kB inactive_file:191584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9276kB shmem:31220kB slab_reclaimable:24908kB slab_unreclaimable:562548kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:52:36 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:52:36 oak-gw06 kernel: Node 0 Normal free:1080496kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6707724kB inactive_file:396804kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44360kB shmem:164984kB slab_reclaimable:264648kB slab_unreclaimable:4374552kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:52:36 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:52:36 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:52:36 oak-gw06 kernel: Node 0 DMA32: 14799*4kB (UEM) 13177*8kB (UEM) 6927*16kB (UEM) 11618*32kB (UEM) 7155*64kB (UEM) 1678*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1359860kB Aug 24 06:52:36 oak-gw06 kernel: Node 0 Normal: 43103*4kB (UEM) 64742*8kB (UEM) 16458*16kB (UEM) 2813*32kB (UEM) 475*64kB (UE) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1080876kB Aug 24 06:52:36 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:52:36 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:52:36 oak-gw06 kernel: 2034043 total pagecache pages Aug 24 06:52:36 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:52:36 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:52:36 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:52:36 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:52:36 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:52:36 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:52:36 oak-gw06 kernel: 127313 pages reserved Aug 24 06:57:36 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:57:36 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:57:36 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:57:36 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:57:36 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97858 ffffffff8168662f Aug 24 06:57:36 oak-gw06 kernel: ffff880172a978e8 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 06:57:36 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a978b8 00000000f1450c91 Aug 24 06:57:36 oak-gw06 kernel: Call Trace: Aug 24 06:57:36 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:57:36 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:57:36 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:57:36 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:57:36 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 06:57:36 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 06:57:36 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:57:36 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:57:36 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:57:36 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:57:36 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:57:36 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:57:36 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:57:36 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:57:36 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:57:36 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:57:36 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:57:36 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:57:36 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:57:36 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:57:36 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:57:36 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:57:36 oak-gw06 kernel: Mem-Info: Aug 24 06:57:36 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1837630 inactive_file:147103 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72389 slab_unreclaimable:1233270#012 mapped:13418 shmem:49051 pagetables:1446 bounce:0#012 free:615132 free_pcp:194 free_cma:0 Aug 24 06:57:36 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:57:36 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:57:36 oak-gw06 kernel: Node 0 DMA32 free:1359956kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:643820kB inactive_file:191584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9272kB shmem:31220kB slab_reclaimable:24908kB slab_unreclaimable:562436kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:57:36 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:57:36 oak-gw06 kernel: Node 0 Normal free:1084328kB min:323104kB low:403880kB high:484656kB active_anon:40192kB inactive_anon:184660kB active_file:6706700kB inactive_file:396828kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44400kB shmem:164984kB slab_reclaimable:264648kB slab_unreclaimable:4370628kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:944kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:57:36 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:57:36 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:57:36 oak-gw06 kernel: Node 0 DMA32: 14803*4kB (UEM) 13177*8kB (UEM) 6936*16kB (UEM) 11618*32kB (UEM) 7155*64kB (UEM) 1678*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1360020kB Aug 24 06:57:36 oak-gw06 kernel: Node 0 Normal: 43304*4kB (UEM) 64794*8kB (UEM) 16620*16kB (UEM) 2825*32kB (UEM) 476*64kB (UE) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1085136kB Aug 24 06:57:36 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:57:36 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:57:36 oak-gw06 kernel: 2033792 total pagecache pages Aug 24 06:57:36 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:57:36 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:57:36 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:57:36 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:57:36 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:57:36 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:57:36 oak-gw06 kernel: 127313 pages reserved Aug 24 06:57:36 oak-gw06 kernel: kworker/u16:0: page allocation failure: order:7, mode:0x80d0 Aug 24 06:57:36 oak-gw06 kernel: CPU: 1 PID: 24083 Comm: kworker/u16:0 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 06:57:36 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 06:57:36 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 06:57:36 oak-gw06 kernel: 00000000000080d0 00000000f1450c91 ffff880172a97808 ffffffff8168662f Aug 24 06:57:36 oak-gw06 kernel: ffff880172a97898 ffffffff81186ba0 ffffffff81189900 0000000000000000 Aug 24 06:57:36 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff880172a97868 00000000f1450c91 Aug 24 06:57:36 oak-gw06 kernel: Call Trace: Aug 24 06:57:36 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 06:57:36 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 06:57:36 oak-gw06 kernel: [] ? drain_pages+0xb0/0xb0 Aug 24 06:57:36 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 06:57:36 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 06:57:36 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 06:57:36 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 06:57:36 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 06:57:36 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 06:57:36 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 06:57:36 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 06:57:36 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 06:57:36 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 06:57:36 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 06:57:36 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 06:57:36 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 06:57:36 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 06:57:36 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 06:57:36 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 06:57:36 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 06:57:36 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 06:57:36 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 06:57:36 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:57:36 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 06:57:36 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 06:57:36 oak-gw06 kernel: Mem-Info: Aug 24 06:57:37 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1837630 inactive_file:147103 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72389 slab_unreclaimable:1233270#012 mapped:13418 shmem:49051 pagetables:1446 bounce:0#012 free:615341 free_pcp:31 free_cma:0 Aug 24 06:57:37 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 06:57:37 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 06:57:37 oak-gw06 kernel: Node 0 DMA32 free:1359956kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:643820kB inactive_file:191584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9272kB shmem:31220kB slab_reclaimable:24908kB slab_unreclaimable:562436kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:57:37 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 06:57:37 oak-gw06 kernel: Node 0 Normal free:1084860kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6706700kB inactive_file:396828kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44400kB shmem:164984kB slab_reclaimable:264648kB slab_unreclaimable:4370628kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:744kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 06:57:37 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 06:57:37 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 06:57:37 oak-gw06 kernel: Node 0 DMA32: 14805*4kB (UEM) 13178*8kB (UEM) 6936*16kB (UEM) 11618*32kB (UEM) 7155*64kB (UEM) 1678*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1360036kB Aug 24 06:57:37 oak-gw06 kernel: Node 0 Normal: 43309*4kB (UEM) 64861*8kB (UEM) 16625*16kB (UEM) 2825*32kB (UEM) 476*64kB (UE) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1085772kB Aug 24 06:57:37 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 06:57:37 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 06:57:37 oak-gw06 kernel: 2033792 total pagecache pages Aug 24 06:57:37 oak-gw06 kernel: 6 pages in swap cache Aug 24 06:57:37 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 06:57:37 oak-gw06 kernel: Free swap = 4193544kB Aug 24 06:57:37 oak-gw06 kernel: Total swap = 4194300kB Aug 24 06:57:37 oak-gw06 kernel: 4194203 pages RAM Aug 24 06:57:37 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 06:57:37 oak-gw06 kernel: 127313 pages reserved Aug 24 07:02:37 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 24 07:02:37 oak-gw06 kernel: CPU: 1 PID: 24242 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 07:02:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 07:02:37 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 07:02:37 oak-gw06 kernel: 00000000000080d0 000000005887b6d3 ffff88010378b858 ffffffff8168662f Aug 24 07:02:37 oak-gw06 kernel: ffff88010378b8e8 ffffffff81186ba0 ffffffff810f9c52 0000000000000010 Aug 24 07:02:37 oak-gw06 kernel: ffffffffffffff80 000080d000000000 0000000000000018 000000005887b6d3 Aug 24 07:02:37 oak-gw06 kernel: Call Trace: Aug 24 07:02:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 07:02:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 07:02:37 oak-gw06 kernel: [] ? on_each_cpu_mask+0x52/0x60 Aug 24 07:02:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 07:02:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 07:02:37 oak-gw06 kernel: [] dma_generic_alloc_coherent+0x8f/0x140 Aug 24 07:02:37 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x21/0x50 Aug 24 07:02:37 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 07:02:37 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 07:02:37 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 07:02:37 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 07:02:37 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 07:02:37 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 07:02:37 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 07:02:37 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 07:02:37 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 07:02:37 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 07:02:37 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 07:02:37 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 07:02:37 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 07:02:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 07:02:37 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 07:02:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 07:02:37 oak-gw06 kernel: Mem-Info: Aug 24 07:02:37 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1837454 inactive_file:147106 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72389 slab_unreclaimable:1231532#012 mapped:13433 shmem:49051 pagetables:1446 bounce:0#012 free:617109 free_pcp:61 free_cma:0 Aug 24 07:02:37 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 07:02:37 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 07:02:37 oak-gw06 kernel: Node 0 DMA32 free:1360844kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:643300kB inactive_file:191584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9276kB shmem:31220kB slab_reclaimable:24908kB slab_unreclaimable:562060kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 07:02:37 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 07:02:37 oak-gw06 kernel: Node 0 Normal free:1090992kB min:323104kB low:403880kB high:484656kB active_anon:40452kB inactive_anon:184660kB active_file:6706516kB inactive_file:396840kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44456kB shmem:164984kB slab_reclaimable:264648kB slab_unreclaimable:4364052kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:256kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 07:02:37 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 07:02:37 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 07:02:37 oak-gw06 kernel: Node 0 DMA32: 14815*4kB (UEM) 13228*8kB (UEM) 6952*16kB (UEM) 11618*32kB (UEM) 7156*64kB (UEM) 1679*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1360924kB Aug 24 07:02:37 oak-gw06 kernel: Node 0 Normal: 43702*4kB (UEM) 64888*8kB (UEM) 16878*16kB (UEM) 2842*32kB (UEM) 477*64kB (UE) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1092216kB Aug 24 07:02:37 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 07:02:37 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 07:02:37 oak-gw06 kernel: 2033556 total pagecache pages Aug 24 07:02:37 oak-gw06 kernel: 6 pages in swap cache Aug 24 07:02:37 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 07:02:37 oak-gw06 kernel: Free swap = 4193544kB Aug 24 07:02:37 oak-gw06 kernel: Total swap = 4194300kB Aug 24 07:02:37 oak-gw06 kernel: 4194203 pages RAM Aug 24 07:02:37 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 07:02:37 oak-gw06 kernel: 127313 pages reserved Aug 24 07:02:37 oak-gw06 kernel: kworker/u16:1: page allocation failure: order:7, mode:0x80d0 Aug 24 07:02:37 oak-gw06 kernel: CPU: 1 PID: 24242 Comm: kworker/u16:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 Aug 24 07:02:37 oak-gw06 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Aug 24 07:02:37 oak-gw06 kernel: Workqueue: rdma_cm cma_work_handler [rdma_cm] Aug 24 07:02:37 oak-gw06 kernel: 00000000000080d0 000000005887b6d3 ffff88010378b808 ffffffff8168662f Aug 24 07:02:37 oak-gw06 kernel: ffff88010378b898 ffffffff81186ba0 0000000000000000 00000000ffffffff Aug 24 07:02:37 oak-gw06 kernel: ffffffffffffff80 000080d000000000 ffff88010378b868 000000005887b6d3 Aug 24 07:02:37 oak-gw06 kernel: Call Trace: Aug 24 07:02:37 oak-gw06 kernel: [] dump_stack+0x19/0x1b Aug 24 07:02:37 oak-gw06 kernel: [] warn_alloc_failed+0x110/0x180 Aug 24 07:02:37 oak-gw06 kernel: [] __alloc_pages_slowpath+0x6b7/0x725 Aug 24 07:02:37 oak-gw06 kernel: [] __alloc_pages_nodemask+0x405/0x420 Aug 24 07:02:37 oak-gw06 kernel: [] alloc_pages_current+0xaa/0x170 Aug 24 07:02:37 oak-gw06 kernel: [] __get_free_pages+0xe/0x50 Aug 24 07:02:37 oak-gw06 kernel: [] swiotlb_alloc_coherent+0x5e/0x150 Aug 24 07:02:37 oak-gw06 kernel: [] x86_swiotlb_alloc_coherent+0x41/0x50 Aug 24 07:02:37 oak-gw06 kernel: [] mlx4_buf_direct_alloc.isra.6+0xd3/0x1a0 [mlx4_core] Aug 24 07:02:37 oak-gw06 kernel: [] mlx4_buf_alloc+0x1cb/0x240 [mlx4_core] Aug 24 07:02:37 oak-gw06 kernel: [] create_qp_common.isra.31+0x5e2/0x1000 [mlx4_ib] Aug 24 07:02:37 oak-gw06 kernel: [] mlx4_ib_create_qp+0x14e/0x470 [mlx4_ib] Aug 24 07:02:37 oak-gw06 kernel: [] ib_create_qp+0x3f/0x250 [ib_core] Aug 24 07:02:37 oak-gw06 kernel: [] rdma_create_qp+0x34/0xb0 [rdma_cm] Aug 24 07:02:37 oak-gw06 kernel: [] kiblnd_create_conn+0xad7/0x1870 [ko2iblnd] Aug 24 07:02:37 oak-gw06 kernel: [] kiblnd_cm_callback+0x1429/0x2290 [ko2iblnd] Aug 24 07:02:37 oak-gw06 kernel: [] cma_work_handler+0x6c/0xa0 [rdma_cm] Aug 24 07:02:37 oak-gw06 kernel: [] process_one_work+0x17b/0x470 Aug 24 07:02:37 oak-gw06 kernel: [] worker_thread+0x126/0x410 Aug 24 07:02:37 oak-gw06 kernel: [] ? rescuer_thread+0x460/0x460 Aug 24 07:02:37 oak-gw06 kernel: [] kthread+0xcf/0xe0 Aug 24 07:02:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 07:02:37 oak-gw06 kernel: [] ret_from_fork+0x58/0x90 Aug 24 07:02:37 oak-gw06 kernel: [] ? kthread_create_on_node+0x140/0x140 Aug 24 07:02:37 oak-gw06 kernel: Mem-Info: Aug 24 07:02:37 oak-gw06 kernel: active_anon:10878 inactive_anon:57055 isolated_anon:0#012 active_file:1837389 inactive_file:147106 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:72389 slab_unreclaimable:1231532#012 mapped:13433 shmem:49051 pagetables:1446 bounce:0#012 free:617257 free_pcp:31 free_cma:0 Aug 24 07:02:37 oak-gw06 kernel: Node 0 DMA free:15892kB min:384kB low:480kB high:576kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 24 07:02:37 oak-gw06 kernel: lowmem_reserve[]: 0 2815 15869 15869 Aug 24 07:02:37 oak-gw06 kernel: Node 0 DMA32 free:1360844kB min:69724kB low:87152kB high:104584kB active_anon:3580kB inactive_anon:43560kB active_file:643300kB inactive_file:191584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129332kB managed:2884592kB mlocked:0kB dirty:0kB writeback:0kB mapped:9276kB shmem:31220kB slab_reclaimable:24908kB slab_unreclaimable:562060kB kernel_stack:1024kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 07:02:37 oak-gw06 kernel: lowmem_reserve[]: 0 0 13053 13053 Aug 24 07:02:37 oak-gw06 kernel: Node 0 Normal free:1092292kB min:323104kB low:403880kB high:484656kB active_anon:39932kB inactive_anon:184660kB active_file:6706256kB inactive_file:396840kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13367060kB mlocked:0kB dirty:0kB writeback:0kB mapped:44456kB shmem:164984kB slab_reclaimable:264648kB slab_unreclaimable:4364052kB kernel_stack:4656kB pagetables:4648kB unstable:0kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 24 07:02:38 oak-gw06 kernel: lowmem_reserve[]: 0 0 0 0 Aug 24 07:02:38 oak-gw06 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Aug 24 07:02:38 oak-gw06 kernel: Node 0 DMA32: 14815*4kB (UEM) 13228*8kB (UEM) 6952*16kB (UEM) 11618*32kB (UEM) 7156*64kB (UEM) 1679*128kB (UEM) 154*256kB (UM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1360924kB Aug 24 07:02:38 oak-gw06 kernel: Node 0 Normal: 43704*4kB (UEM) 64891*8kB (UEM) 16898*16kB (UEM) 2848*32kB (UEM) 477*64kB (UE) 53*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1092760kB Aug 24 07:02:38 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Aug 24 07:02:38 oak-gw06 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 24 07:02:38 oak-gw06 kernel: 2033459 total pagecache pages Aug 24 07:02:38 oak-gw06 kernel: 6 pages in swap cache Aug 24 07:02:38 oak-gw06 kernel: Swap cache stats: add 189, delete 183, find 0/0 Aug 24 07:02:38 oak-gw06 kernel: Free swap = 4193544kB Aug 24 07:02:38 oak-gw06 kernel: Total swap = 4194300kB Aug 24 07:02:38 oak-gw06 kernel: 4194203 pages RAM Aug 24 07:02:38 oak-gw06 kernel: 0 pages HighMem/MovableOnly Aug 24 07:02:38 oak-gw06 kernel: 127313 pages reserved Aug 25 18:44:12 oak-gw06 kernel: Lustre: DEBUG MARKER: Fri Aug 25 18:44:12 2017 Aug 31 11:27:05 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] Aug 31 11:27:05 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) original client csum 4ecd330 (type 1), server csum 5610e5e5 (type 1), client csum now 4ecd330 Aug 31 11:27:05 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88001b654600 x1566273879372336/t60132831973(60132831973) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504204071 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:27:06 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] Aug 31 11:27:06 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum 4ecd330 (type 1), server csum 5610e5e5 (type 1), client csum now 4ecd330 Aug 31 11:27:06 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802b770ed00 x1566273879372416/t60132831982(60132831982) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504204073 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:27:08 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] Aug 31 11:27:08 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum 4ecd330 (type 1), server csum 5610e5e5 (type 1), client csum now 4ecd330 Aug 31 11:27:08 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800683c0300 x1566273879373872/t60132832002(60132832002) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504204075 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:27:11 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] Aug 31 11:27:11 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum 4ecd330 (type 1), server csum 5610e5e5 (type 1), client csum now 4ecd330 Aug 31 11:27:11 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800683c0600 x1566273879375136/t60132832026(60132832026) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504204078 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:27:21 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] Aug 31 11:27:21 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 31 11:27:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) original client csum 4ecd330 (type 1), server csum 5610e5e5 (type 1), client csum now 4ecd330 Aug 31 11:27:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) Skipped 1 previous similar message Aug 31 11:27:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803b9792100 x1566273879378544/t60132832103(60132832103) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504204087 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:27:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 1 previous similar message Aug 31 11:27:34 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] Aug 31 11:27:34 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 31 11:27:34 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum 4ecd330 (type 1), server csum 5610e5e5 (type 1), client csum now 4ecd330 Aug 31 11:27:34 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 1 previous similar message Aug 31 11:27:34 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88041b857000 x1566273879385120/t60132832218(60132832218) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504204101 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:27:34 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 1 previous similar message Aug 31 11:27:51 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] Aug 31 11:27:51 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 31 11:27:51 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum 4ecd330 (type 1), server csum 5610e5e5 (type 1), client csum now 4ecd330 Aug 31 11:27:51 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 1 previous similar message Aug 31 11:27:51 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88001b655b00 x1566273879393600/t60132832445(60132832445) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504204118 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:27:51 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 1 previous similar message Aug 31 11:28:01 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST001a-osc-ffff88041b99c000: too many resent retries for object: 0:4413301, rc = -11. Aug 31 11:32:35 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [2963800064-2967470079] Aug 31 11:32:35 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 31 11:32:35 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum 46fb7986 (type 1), server csum 557743ef (type 1), client csum now 46fb7986 Aug 31 11:32:35 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 1 previous similar message Aug 31 11:32:35 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880367f21500 x1566273879618992/t60132835521(60132835521) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504204402 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:33:27 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST001a-osc-ffff88041b99c000: too many resent retries for object: 0:4413301, rc = -11. Aug 31 11:37:42 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:37:42 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:37:42 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880331a91800 x1566273879895088/t0(0) o3->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504204710 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 11:37:42 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Aug 31 11:37:43 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:37:43 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:37:45 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:37:45 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:37:48 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:37:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:37:52 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:37:52 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:38:03 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:38:03 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 31 11:38:03 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:38:03 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 1 previous similar message Aug 31 11:38:27 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:38:27 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 11:38:27 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:38:27 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Aug 31 11:38:37 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001a-osc-ffff88041b99c000: too many resent retries for object: 0:4413301, rc = -11. Aug 31 11:39:05 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:39:05 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Aug 31 11:39:05 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:39:05 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Aug 31 11:39:32 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST001a-osc-ffff88041b99c000: too many resent retries for object: 0:4413301, rc = -11. Aug 31 11:39:55 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880331a92d00 x1566273880509696/t0(0) o3->oak-OST000a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504204842 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 11:39:55 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 19 previous similar messages Aug 31 11:40:10 oak-gw06 kernel: LustreError: 133-1: oak-OST000a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4409505 extent [29903290368-29904338943] Aug 31 11:40:10 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Aug 31 11:40:10 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 5fbc7f37, server 46e0c8d, cksum_type 1 Aug 31 11:40:10 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Aug 31 11:40:50 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST000a-osc-ffff88041b99c000: too many resent retries for object: 0:4409505, rc = -11. Aug 31 11:41:45 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST000a-osc-ffff88041b99c000: too many resent retries for object: 0:4409505, rc = -11. Aug 31 11:42:21 oak-gw06 kernel: LustreError: 133-1: oak-OST000a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4409505 extent [29903290368-29904338943] Aug 31 11:42:21 oak-gw06 kernel: LustreError: Skipped 24 previous similar messages Aug 31 11:42:21 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client 5fbc7f37, server 46e0c8d, cksum_type 1 Aug 31 11:42:21 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 24 previous similar messages Aug 31 11:42:40 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST000a-osc-ffff88041b99c000: too many resent retries for object: 0:4409505, rc = -11. Aug 31 11:58:48 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b28:0x0] object 0x0:3755960 extent [61770301440-61771350015] Aug 31 11:58:48 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Aug 31 11:58:48 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) original client csum ddce27b3 (type 1), server csum 34f06c50 (type 1), client csum now ddce27b3 Aug 31 11:58:48 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Aug 31 11:58:48 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880136f32700 x1566273880943232/t51542778710(51542778710) o4->oak-OST001e-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504205939 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:58:48 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Aug 31 11:58:58 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b28:0x0] object 0x0:3755960 extent [61770301440-61771350015] Aug 31 11:58:58 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Aug 31 11:58:58 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) original client csum ddce27b3 (type 1), server csum 34f06c50 (type 1), client csum now ddce27b3 Aug 31 11:58:58 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Aug 31 11:59:16 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b28:0x0] object 0x0:3755960 extent [61770301440-61771350015] Aug 31 11:59:16 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 11:59:16 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum ddce27b3 (type 1), server csum 34f06c50 (type 1), client csum now ddce27b3 Aug 31 11:59:16 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 11:59:24 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88012821e100 x1566273880956800/t51542778871(51542778871) o4->oak-OST001e-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504206013 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:59:24 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 7 previous similar messages Aug 31 11:59:43 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001e-osc-ffff88041b99c000: too many resent retries for object: 0:3755960, rc = -11. Aug 31 12:00:26 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b28:0x0] object 0x0:3755960 extent [62205722624-62206771199] Aug 31 12:00:26 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Aug 31 12:00:26 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum b00ad904 (type 1), server csum af5a1186 (type 1), client csum now b00ad904 Aug 31 12:00:26 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Aug 31 12:00:29 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880125c4d800 x1566273880998512/t51542779207(51542779207) o4->oak-OST001e-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504206078 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 12:00:29 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 3 previous similar messages Aug 31 12:01:21 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST001e-osc-ffff88041b99c000: too many resent retries for object: 0:3755960, rc = -11. Aug 31 12:01:51 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b56:0x0] object 0x0:4397797 extent [8915517440-8917876735] Aug 31 12:01:51 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Aug 31 12:01:51 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) original client csum 6e00acc (type 1), server csum 1bd6cfbe (type 1), client csum now 6e00acc Aug 31 12:01:51 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Aug 31 12:02:46 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0006-osc-ffff88041b99c000: too many resent retries for object: 0:4397797, rc = -11. Aug 31 12:43:04 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f83:0x15488:0x0] object 0x0:3991245 extent [15734931456-15737028607] Aug 31 12:43:04 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Aug 31 12:43:04 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum c1563535 (type 1), server csum dbb78289 (type 1), client csum now c1563535 Aug 31 12:43:04 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Aug 31 12:43:04 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880397cdd200 x1566273885951680/t51542723909(51542723909) o4->oak-OST0023-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504208631 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 12:43:04 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 17 previous similar messages Aug 31 12:43:25 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f83:0x15488:0x0] object 0x0:3991245 extent [15734931456-15737028607] Aug 31 12:43:25 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 31 12:43:25 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum c1563535 (type 1), server csum dbb78289 (type 1), client csum now c1563535 Aug 31 12:43:25 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 5 previous similar messages Aug 31 12:43:25 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803c22dd800 x1566273885958448/t51542724081(51542724081) o4->oak-OST0023-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504208652 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 12:43:25 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Aug 31 12:43:59 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f83:0x15488:0x0] object 0x0:3991245 extent [15734931456-15737028607] Aug 31 12:43:59 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Aug 31 12:43:59 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum c1563535 (type 1), server csum dbb78289 (type 1), client csum now c1563535 Aug 31 12:43:59 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Aug 31 12:43:59 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0023-osc-ffff88041b99c000: too many resent retries for object: 0:3991245, rc = -11. Aug 31 12:44:22 oak-gw06 kernel: LustreError: 133-1: oak-OST0004-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:552911 extent [39845888-40894463] Aug 31 12:44:22 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 12:44:22 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client a5db2e8a, server 608d44ea, cksum_type 1 Aug 31 12:44:22 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Aug 31 12:44:22 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801aeb23900 x1566273886000624/t0(0) o3->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504208709 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 12:44:22 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 3 previous similar messages Aug 31 12:44:58 oak-gw06 kernel: LustreError: 133-1: oak-OST0004-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:552911 extent [39845888-40894463] Aug 31 12:44:58 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Aug 31 12:44:58 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client a5db2e8a, server 608d44ea, cksum_type 1 Aug 31 12:44:58 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Aug 31 12:45:17 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0004-osc-ffff88041b99c000: too many resent retries for object: 0:552911, rc = -11. Aug 31 12:45:27 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88039fc8b900 x1566273886044896/t0(0) o3->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504208774 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 12:45:27 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 13 previous similar messages Aug 31 12:46:12 oak-gw06 kernel: LustreError: 133-1: oak-OST0004-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:552911 extent [39845888-40894463] Aug 31 12:46:12 oak-gw06 kernel: LustreError: Skipped 12 previous similar messages Aug 31 12:46:12 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client a5db2e8a, server 608d44ea, cksum_type 1 Aug 31 12:46:12 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 12 previous similar messages Aug 31 12:46:12 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0004-osc-ffff88041b99c000: too many resent retries for object: 0:552911, rc = -11. Aug 31 12:47:07 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0004-osc-ffff88041b99c000: too many resent retries for object: 0:552911, rc = -11. Aug 31 12:47:43 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88020afe7600 x1566273886122000/t0(0) o3->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504208874 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 12:47:43 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 23 previous similar messages Aug 31 12:48:02 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0004-osc-ffff88041b99c000: too many resent retries for object: 0:552911, rc = -11. Aug 31 12:48:23 oak-gw06 kernel: LustreError: 133-1: oak-OST0004-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:552911 extent [39845888-40894463] Aug 31 12:48:23 oak-gw06 kernel: LustreError: Skipped 28 previous similar messages Aug 31 12:48:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client a5db2e8a, server 608d44ea, cksum_type 1 Aug 31 12:48:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 28 previous similar messages Aug 31 12:48:57 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0004-osc-ffff88041b99c000: too many resent retries for object: 0:552911, rc = -11. Aug 31 13:02:24 oak-gw06 kernel: LustreError: 133-1: oak-OST001d-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:239355 extent [27262976-28311551] Aug 31 13:02:24 oak-gw06 kernel: LustreError: Skipped 4 previous similar messages Aug 31 13:02:24 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client 2bd21bed, server 7ca5c1f2, cksum_type 1 Aug 31 13:02:24 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 4 previous similar messages Aug 31 13:02:24 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802b770e700 x1566273887994352/t0(0) o3->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504209753 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 13:02:24 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 11 previous similar messages Aug 31 13:03:00 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8804138d0000 x1566273888009536/t0(0) o3->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504209789 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 13:03:00 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 7 previous similar messages Aug 31 13:03:16 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:239355, rc = -11. Aug 31 13:04:08 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:239355, rc = -11. Aug 31 13:04:08 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801e7da0300 x1566273888066080/t0(0) o3->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504209857 ref 2 fl Interpret:RM/0/0 rc 700416/700416 Aug 31 13:04:08 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 11 previous similar messages Aug 31 13:05:00 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:239355, rc = -11. Aug 31 13:05:52 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:239355, rc = -11. Aug 31 13:06:21 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88022563cc00 x1566273888132288/t0(0) o3->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 624/400 e 0 to 0 dl 1504209990 ref 2 fl Interpret:RM/0/0 rc 704512/704512 Aug 31 13:06:21 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 26 previous similar messages Aug 31 13:07:37 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:239355, rc = -11. Aug 31 13:07:37 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 13:52:51 oak-gw06 kernel: LustreError: 133-1: oak-OST0019-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:239998 extent [812646400-813694975] Aug 31 13:52:51 oak-gw06 kernel: LustreError: Skipped 65 previous similar messages Aug 31 13:52:51 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 48e11765, server f3e3cbdc, cksum_type 1 Aug 31 13:52:51 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 65 previous similar messages Aug 31 13:52:51 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880101731800 x1566273895440048/t0(0) o3->oak-OST0019-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504212823 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 13:52:51 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 12 previous similar messages Aug 31 13:53:27 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8804138d1500 x1566273895483904/t0(0) o3->oak-OST0019-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504212859 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 13:53:27 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 7 previous similar messages Aug 31 13:53:46 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0019-osc-ffff88041b99c000: too many resent retries for object: 0:239998, rc = -11. Aug 31 13:53:56 oak-gw06 kernel: LustreError: 133-1: oak-OST0019-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:239998 extent [812646400-813694975] Aug 31 13:53:56 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Aug 31 13:53:56 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 48e11765, server f3e3cbdc, cksum_type 1 Aug 31 13:53:56 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 14 previous similar messages Aug 31 13:54:31 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880044d02100 x1566273895762752/t0(0) o3->oak-OST0019-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504212923 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 13:54:31 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 10 previous similar messages Aug 31 13:54:41 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0019-osc-ffff88041b99c000: too many resent retries for object: 0:239998, rc = -11. Aug 31 13:55:36 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0019-osc-ffff88041b99c000: too many resent retries for object: 0:239998, rc = -11. Aug 31 13:57:05 oak-gw06 kernel: LustreError: 133-1: oak-OST0011-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371203 extent [940572672-941621247] Aug 31 13:57:05 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Aug 31 13:57:05 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 9c853d33, server ec07fbc5, cksum_type 1 Aug 31 13:57:05 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Aug 31 13:57:05 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88029be14000 x1566273896455872/t0(0) o3->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504213036 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 13:57:05 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 10 previous similar messages Aug 31 13:58:00 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0011-osc-ffff88041b99c000: too many resent retries for object: 0:371203, rc = -11. Aug 31 14:18:07 oak-gw06 kernel: LustreError: 133-1: oak-OST0012-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321391 extent [1389363200-1390411775] Aug 31 14:18:07 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Aug 31 14:18:07 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 829abbfd, server 1291d7d7, cksum_type 1 Aug 31 14:18:07 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Aug 31 14:18:07 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801be08c000 x1566273899891344/t0(0) o3->oak-OST0012-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504214334 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 14:18:07 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Aug 31 14:18:43 oak-gw06 kernel: LustreError: 133-1: oak-OST0012-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321391 extent [1389363200-1390411775] Aug 31 14:18:43 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Aug 31 14:18:43 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 829abbfd, server 1291d7d7, cksum_type 1 Aug 31 14:18:43 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Aug 31 14:18:43 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801d2f1d500 x1566273899988832/t0(0) o3->oak-OST0012-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504214370 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 14:18:43 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 7 previous similar messages Aug 31 14:19:02 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0012-osc-ffff88041b99c000: too many resent retries for object: 0:321391, rc = -11. Aug 31 14:19:02 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 14:19:47 oak-gw06 kernel: LustreError: 133-1: oak-OST0012-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321391 extent [1389363200-1390411775] Aug 31 14:19:47 oak-gw06 kernel: LustreError: Skipped 11 previous similar messages Aug 31 14:19:47 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 829abbfd, server 1291d7d7, cksum_type 1 Aug 31 14:19:47 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 11 previous similar messages Aug 31 14:19:47 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880399575b00 x1566273900306256/t0(0) o3->oak-OST0012-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504214434 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 14:19:47 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 10 previous similar messages Aug 31 14:19:57 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0012-osc-ffff88041b99c000: too many resent retries for object: 0:321391, rc = -11. Aug 31 14:20:52 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0012-osc-ffff88041b99c000: too many resent retries for object: 0:321391, rc = -11. Aug 31 14:21:59 oak-gw06 kernel: LustreError: 133-1: oak-OST0012-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321391 extent [493879296-494927871] Aug 31 14:21:59 oak-gw06 kernel: LustreError: Skipped 27 previous similar messages Aug 31 14:21:59 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client db927ccb, server d71cb979, cksum_type 1 Aug 31 14:21:59 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 27 previous similar messages Aug 31 14:21:59 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88012cc8b600 x1566273900660736/t0(0) o3->oak-OST0012-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504214566 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 14:21:59 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 24 previous similar messages Aug 31 14:22:44 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0012-osc-ffff88041b99c000: too many resent retries for object: 0:321391, rc = -11. Aug 31 14:22:44 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 14:38:19 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321360 extent [864026624-865075199] Aug 31 14:38:19 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Aug 31 14:38:19 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 8ce62d65, server fda437a3, cksum_type 1 Aug 31 14:38:19 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Aug 31 14:38:19 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800697c9800 x1566273904699184/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504215506 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 14:38:19 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Aug 31 14:38:54 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321360 extent [864026624-865075199] Aug 31 14:38:54 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Aug 31 14:38:54 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 8ce62d65, server fda437a3, cksum_type 1 Aug 31 14:38:54 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Aug 31 14:38:54 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800697cb300 x1566273904717632/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504215542 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 14:38:54 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 7 previous similar messages Aug 31 14:39:08 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321360, rc = -11. Aug 31 14:39:08 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 14:39:57 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321360, rc = -11. Aug 31 15:14:48 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321374 extent [208666624-209715199] Aug 31 15:14:48 oak-gw06 kernel: LustreError: Skipped 13 previous similar messages Aug 31 15:14:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 5920f5c6, server b566e40b, cksum_type 1 Aug 31 15:14:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 13 previous similar messages Aug 31 15:14:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b9cf3600 x1566273911281312/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504217699 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:14:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 11 previous similar messages Aug 31 15:14:58 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321374 extent [208666624-209715199] Aug 31 15:14:58 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Aug 31 15:14:58 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 5920f5c6, server b566e40b, cksum_type 1 Aug 31 15:14:58 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 3 previous similar messages Aug 31 15:14:58 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88041364ad00 x1566273911284064/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504217709 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:14:58 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 3 previous similar messages Aug 31 15:15:16 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321374 extent [208666624-209715199] Aug 31 15:15:16 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 15:15:16 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client 5920f5c6, server b566e40b, cksum_type 1 Aug 31 15:15:16 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Aug 31 15:15:16 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801d8886a00 x1566273911291008/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504217727 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:15:16 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 2 previous similar messages Aug 31 15:15:43 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321374, rc = -11. Aug 31 15:15:49 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321374 extent [208666624-209715199] Aug 31 15:15:49 oak-gw06 kernel: LustreError: Skipped 6 previous similar messages Aug 31 15:15:49 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 5920f5c6, server b566e40b, cksum_type 1 Aug 31 15:15:49 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 6 previous similar messages Aug 31 15:15:49 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802ad776a00 x1566273911455248/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504217760 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:15:49 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Aug 31 15:16:38 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321374, rc = -11. Aug 31 15:16:53 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321374 extent [208666624-209715199] Aug 31 15:16:53 oak-gw06 kernel: LustreError: Skipped 12 previous similar messages Aug 31 15:16:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 5920f5c6, server b566e40b, cksum_type 1 Aug 31 15:16:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 12 previous similar messages Aug 31 15:16:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803f62cea00 x1566273911581200/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504217824 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:16:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 11 previous similar messages Aug 31 15:17:33 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321374, rc = -11. Aug 31 15:18:30 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321374, rc = -11. Aug 31 15:19:06 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321374 extent [208666624-209387519] Aug 31 15:19:06 oak-gw06 kernel: LustreError: Skipped 24 previous similar messages Aug 31 15:19:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 5e93980e, server d9790f9d, cksum_type 1 Aug 31 15:19:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 24 previous similar messages Aug 31 15:19:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88038a226400 x1566273911828496/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504217957 ref 2 fl Interpret:RM/0/0 rc 720896/720896 Aug 31 15:19:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 22 previous similar messages Aug 31 15:19:25 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321374, rc = -11. Aug 31 15:25:15 oak-gw06 kernel: LustreError: 133-1: oak-OST001d-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240031 extent [455081984-456130559] Aug 31 15:25:15 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 15:25:15 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 9ba42b9d, server 64849ffc, cksum_type 1 Aug 31 15:25:15 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Aug 31 15:25:15 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800a2f6e100 x1566273912549776/t0(0) o3->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504218326 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:25:15 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 1 previous similar message Aug 31 15:26:10 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:240031, rc = -11. Aug 31 15:28:56 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:240031, rc = -11. Aug 31 15:28:56 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 15:38:20 oak-gw06 kernel: LustreError: 133-1: oak-OST001d-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240033 extent [52428800-53477375] Aug 31 15:38:21 oak-gw06 kernel: LustreError: Skipped 65 previous similar messages Aug 31 15:38:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 107e0d8d, server d06f4de0, cksum_type 1 Aug 31 15:38:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 65 previous similar messages Aug 31 15:38:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880087db7900 x1566273914409648/t0(0) o3->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504219111 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:38:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 59 previous similar messages Aug 31 15:39:16 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:240033, rc = -11. Aug 31 15:39:16 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 15:44:32 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6c99:0x0] object 0x0:2281084 extent [11516772352-11517820927] Aug 31 15:44:32 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum 4c6ff3eb (type 1), server csum 671924f5 (type 1), client csum now 4c6ff3eb Aug 31 15:44:42 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6c99:0x0] object 0x0:2281084 extent [11516772352-11517820927] Aug 31 15:44:42 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Aug 31 15:44:42 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) original client csum 4c6ff3eb (type 1), server csum 671924f5 (type 1), client csum now 4c6ff3eb Aug 31 15:44:42 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Aug 31 15:45:00 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6c99:0x0] object 0x0:2281084 extent [11516772352-11517820927] Aug 31 15:45:00 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 15:45:00 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) original client csum 4c6ff3eb (type 1), server csum 671924f5 (type 1), client csum now 4c6ff3eb Aug 31 15:45:00 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 15:48:05 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:553509, rc = -11. Aug 31 15:48:05 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Aug 31 15:48:21 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:553509 extent [20971520-22020095] Aug 31 15:48:21 oak-gw06 kernel: LustreError: Skipped 59 previous similar messages Aug 31 15:48:21 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client ee489532, server d6c6690a, cksum_type 1 Aug 31 15:48:21 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 59 previous similar messages Aug 31 15:48:21 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880406f4c600 x1566273915646768/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504219713 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:48:21 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 64 previous similar messages Aug 31 16:12:31 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:553516 extent [428867584-429916159] Aug 31 16:12:31 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 31 16:12:31 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client e47a92c6, server c71a4ec1, cksum_type 1 Aug 31 16:12:31 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Aug 31 16:12:31 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88024212bc00 x1566273920953264/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504221198 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 16:12:31 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 4 previous similar messages Aug 31 16:13:26 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:553516, rc = -11. Aug 31 16:13:26 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 16:13:47 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:553516 extent [428867584-429916159] Aug 31 16:13:47 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Aug 31 16:13:47 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client e47a92c6, server c71a4ec1, cksum_type 1 Aug 31 16:13:47 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Aug 31 16:13:47 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802fe512a00 x1566273921133664/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504221274 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 16:13:47 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Aug 31 16:15:16 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:553516, rc = -11. Aug 31 16:15:16 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 16:16:21 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:553516 extent [133169152-134217727] Aug 31 16:16:21 oak-gw06 kernel: LustreError: Skipped 30 previous similar messages Aug 31 16:16:21 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 93f3c597, server 84a62819, cksum_type 1 Aug 31 16:16:21 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 30 previous similar messages Aug 31 16:16:21 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880118ee1500 x1566273921589680/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504221428 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 16:16:21 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 27 previous similar messages Aug 31 16:18:01 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:553516, rc = -11. Aug 31 16:18:01 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 16:21:26 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4381453 extent [2012217344-2013265919] Aug 31 16:21:26 oak-gw06 kernel: LustreError: Skipped 27 previous similar messages Aug 31 16:21:26 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client cf8c733, server 843ede42, cksum_type 1 Aug 31 16:21:26 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 27 previous similar messages Aug 31 16:21:26 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880062d9fc00 x1566273922064976/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504221697 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 16:21:26 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 25 previous similar messages Aug 31 16:23:16 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:4381453, rc = -11. Aug 31 16:23:16 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 16:35:33 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:239955 extent [186646528-187695103] Aug 31 16:35:33 oak-gw06 kernel: LustreError: Skipped 22 previous similar messages Aug 31 16:35:33 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client 55cace85, server 267d4825, cksum_type 1 Aug 31 16:35:33 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 22 previous similar messages Aug 31 16:35:33 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800259e8900 x1566273926180160/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504222580 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 16:35:33 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 19 previous similar messages Aug 31 16:36:28 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:239955, rc = -11. Aug 31 16:48:09 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:239960 extent [169869312-170917887] Aug 31 16:48:09 oak-gw06 kernel: LustreError: Skipped 65 previous similar messages Aug 31 16:48:09 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 8857b49c, server 291f7e3a, cksum_type 1 Aug 31 16:48:09 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 65 previous similar messages Aug 31 16:48:09 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800a2f6e400 x1566273928663216/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504223300 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 16:48:09 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 59 previous similar messages Aug 31 16:49:04 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:239960, rc = -11. Aug 31 16:49:04 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) Skipped 5 previous similar messages Aug 31 16:58:52 oak-gw06 kernel: LustreError: 133-1: oak-OST0011-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371483 extent [1670381568-1671430143] Aug 31 16:58:52 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Aug 31 16:58:52 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client 17c9517, server 80c262, cksum_type 1 Aug 31 16:58:52 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Aug 31 16:58:52 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88030f8c0c00 x1566273931461680/t0(0) o3->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504223979 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 16:58:52 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 49 previous similar messages Aug 31 16:59:47 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0011-osc-ffff88041b99c000: too many resent retries for object: 0:371483, rc = -11. Aug 31 16:59:47 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Aug 31 17:11:22 oak-gw06 kernel: LustreError: 133-1: oak-OST0011-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371478 extent [163577856-164626431] Aug 31 17:11:22 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Aug 31 17:11:22 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client a31d70ca, server ad590d5f, cksum_type 1 Aug 31 17:11:22 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Aug 31 17:11:22 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800ba229b00 x1566273934115664/t0(0) o3->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504224694 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 17:11:22 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Aug 31 17:12:17 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST0011-osc-ffff88041b99c000: too many resent retries for object: 0:371478, rc = -11. Aug 31 17:12:17 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 17:24:39 oak-gw06 kernel: LustreError: 133-1: oak-OST0011-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371485 extent [187695104-188743679] Aug 31 17:24:39 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Aug 31 17:24:39 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client f53a1d0f, server 9b5948f8, cksum_type 1 Aug 31 17:24:39 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Aug 31 17:24:39 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88010eba0600 x1566273936942048/t0(0) o3->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504225491 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 17:24:39 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Aug 31 17:25:34 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0011-osc-ffff88041b99c000: too many resent retries for object: 0:371485, rc = -11. Aug 31 17:25:34 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 17:43:15 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:239963 extent [233832448-234881023] Aug 31 17:43:15 oak-gw06 kernel: LustreError: Skipped 65 previous similar messages Aug 31 17:43:15 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client 2b3a1356, server 7aec27ac, cksum_type 1 Aug 31 17:43:15 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 65 previous similar messages Aug 31 17:43:15 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802159fd500 x1566273942693856/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504226607 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 17:43:15 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 59 previous similar messages Aug 31 17:44:10 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:239963, rc = -11. Aug 31 17:44:10 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 5 previous similar messages Aug 31 18:04:24 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240001 extent [191889408-192937983] Aug 31 18:04:24 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Aug 31 18:04:24 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 6a683663, server e9390f02, cksum_type 1 Aug 31 18:04:24 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Aug 31 18:04:24 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802cc5aa700 x1566273947601424/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504227896 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 18:04:24 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 49 previous similar messages Aug 31 18:05:19 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:240001, rc = -11. Aug 31 18:05:19 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Aug 31 18:05:40 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240001 extent [191889408-192557055] Aug 31 18:05:40 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Aug 31 18:05:40 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client 8cabe409, server c8c3057f, cksum_type 1 Aug 31 18:05:40 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Aug 31 18:05:40 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88033a3e1800 x1566273947872320/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504227972 ref 2 fl Interpret:RM/0/0 rc 667648/667648 Aug 31 18:05:40 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Aug 31 18:07:09 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:240001, rc = -11. Aug 31 18:07:09 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 18:08:11 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240001 extent [191889408-192937983] Aug 31 18:08:11 oak-gw06 kernel: LustreError: Skipped 29 previous similar messages Aug 31 18:08:11 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 6a683663, server e9390f02, cksum_type 1 Aug 31 18:08:11 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 29 previous similar messages Aug 31 18:08:11 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880118623c00 x1566273948389072/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504228123 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 18:08:11 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 26 previous similar messages Aug 31 18:13:15 oak-gw06 kernel: LustreError: 133-1: oak-OST0011-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371527 extent [68157440-69206015] Aug 31 18:13:15 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Aug 31 18:13:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client fd3c2e16, server 26920f47, cksum_type 1 Aug 31 18:13:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Aug 31 18:13:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0011-osc-ffff88041b99c000: too many resent retries for object: 0:371527, rc = -11. Aug 31 18:13:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 18:13:15 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880193933600 x1566273949963840/t0(0) o3->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504228418 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 18:13:15 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 16 previous similar messages Aug 31 18:21:17 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0011-osc-ffff88041b99c000: too many resent retries for object: 0:371531, rc = -11. Aug 31 18:21:17 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Aug 31 18:23:18 oak-gw06 kernel: LustreError: 133-1: oak-OST0011-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371531 extent [286261248-287309823] Aug 31 18:23:18 oak-gw06 kernel: LustreError: Skipped 81 previous similar messages Aug 31 18:23:18 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 64058ff4, server 3319c8df, cksum_type 1 Aug 31 18:23:18 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 81 previous similar messages Aug 31 18:23:18 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800855fad00 x1566273951269200/t0(0) o3->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504229009 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 18:23:18 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 73 previous similar messages Aug 31 18:40:39 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6905:0x0] object 0x0:3989770 extent [800849920-801898495] Aug 31 18:40:39 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Aug 31 18:40:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum e8f92351 (type 1), server csum 7403364b (type 1), client csum now e8f92351 Aug 31 18:40:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Aug 31 18:40:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880425e39200 x1566273954643936/t51542881734(51542881734) o4->oak-OST0023-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504230051 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 18:40:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Aug 31 18:40:45 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6905:0x0] object 0x0:3989770 extent [800849920-801898495] Aug 31 18:40:45 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 18:40:45 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum e8f92351 (type 1), server csum 7403364b (type 1), client csum now e8f92351 Aug 31 18:40:45 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 18:40:54 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6905:0x0] object 0x0:3989770 extent [800849920-801898495] Aug 31 18:40:54 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 31 18:40:54 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum e8f92351 (type 1), server csum 7403364b (type 1), client csum now e8f92351 Aug 31 18:40:54 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 1 previous similar message Aug 31 18:41:15 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6905:0x0] object 0x0:3989770 extent [800849920-801898495] Aug 31 18:41:15 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 18:41:15 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum e8f92351 (type 1), server csum 7403364b (type 1), client csum now e8f92351 Aug 31 18:41:15 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 18:41:34 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0023-osc-ffff88041b99c000: too many resent retries for object: 0:3989770, rc = -11. Aug 31 18:41:34 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Aug 31 18:42:04 oak-gw06 kernel: LustreError: 133-1: oak-OST0019-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240120 extent [891289600-892338175] Aug 31 18:42:04 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Aug 31 18:42:04 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client 56aaab50, server 38349f19, cksum_type 1 Aug 31 18:42:04 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Aug 31 18:42:59 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST0019-osc-ffff88041b99c000: too many resent retries for object: 0:240120, rc = -11. Aug 31 18:45:46 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0019-osc-ffff88041b99c000: too many resent retries for object: 0:240120, rc = -11. Aug 31 18:45:46 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 18:54:53 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:321555 extent [1815085056-1816133631] Aug 31 18:54:53 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Aug 31 18:54:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 9d48c84a, server 35359b11, cksum_type 1 Aug 31 18:54:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Aug 31 18:54:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880252796700 x1566273957383536/t0(0) o3->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504230904 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 18:54:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 59 previous similar messages Aug 31 18:55:48 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0017-osc-ffff88041b99c000: too many resent retries for object: 0:321555, rc = -11. Aug 31 18:55:48 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 19:06:12 oak-gw06 kernel: LustreError: 133-1: oak-OST0014-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321526 extent [1475346432-1476395007] Aug 31 19:06:12 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Aug 31 19:06:12 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 2e9f5d89, server 23ecd3d1, cksum_type 1 Aug 31 19:06:12 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Aug 31 19:06:12 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b9deb600 x1566273959345360/t0(0) o3->oak-OST0014-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504231619 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 19:06:12 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Aug 31 19:07:07 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0014-osc-ffff88041b99c000: too many resent retries for object: 0:321526, rc = -11. Aug 31 19:07:07 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 19:20:15 oak-gw06 kernel: LustreError: 133-1: oak-OST0000-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:553502 extent [420478976-421527551] Aug 31 19:20:15 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Aug 31 19:20:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 614a0826, server 451c4910, cksum_type 1 Aug 31 19:20:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Aug 31 19:20:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800563c7300 x1566273962459072/t0(0) o3->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504232426 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 19:20:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Aug 31 19:21:10 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0000-osc-ffff88041b99c000: too many resent retries for object: 0:553502, rc = -11. Aug 31 19:21:10 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 19:32:33 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240001 extent [164626432-165675007] Aug 31 19:32:33 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Aug 31 19:32:33 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client bbd7793b, server 1c573201, cksum_type 1 Aug 31 19:32:33 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Aug 31 19:32:33 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803146af300 x1566273965313440/t0(0) o3->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504233165 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 19:32:33 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 49 previous similar messages Aug 31 19:33:28 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST001a-osc-ffff88041b99c000: too many resent retries for object: 0:240001, rc = -11. Aug 31 19:33:28 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Aug 31 20:10:43 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371582 extent [1054867456-1055916031] Aug 31 20:10:43 oak-gw06 kernel: LustreError: Skipped 65 previous similar messages Aug 31 20:10:43 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client a0a17d83, server b6d0d647, cksum_type 1 Aug 31 20:10:43 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 65 previous similar messages Aug 31 20:10:43 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8804147de700 x1566273971034784/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504235490 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:10:43 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 59 previous similar messages Aug 31 20:11:38 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0007-osc-ffff88041b99c000: too many resent retries for object: 0:371582, rc = -11. Aug 31 20:11:38 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 5 previous similar messages Aug 31 20:11:59 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371582 extent [1054867456-1055916031] Aug 31 20:11:59 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Aug 31 20:11:59 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client a0a17d83, server b6d0d647, cksum_type 1 Aug 31 20:11:59 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Aug 31 20:11:59 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b712b900 x1566273971272112/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504235531 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:11:59 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Aug 31 20:13:28 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0007-osc-ffff88041b99c000: too many resent retries for object: 0:371582, rc = -11. Aug 31 20:13:28 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 20:30:02 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371577 extent [143654912-144703487] Aug 31 20:30:02 oak-gw06 kernel: LustreError: Skipped 15 previous similar messages Aug 31 20:30:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client 305d2dcf, server f98e3eb, cksum_type 1 Aug 31 20:30:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 15 previous similar messages Aug 31 20:30:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802b770f600 x1566273973054240/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504236613 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:30:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 13 previous similar messages Aug 31 20:30:23 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371577 extent [143654912-144703487] Aug 31 20:30:23 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 31 20:30:23 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client 305d2dcf, server f98e3eb, cksum_type 1 Aug 31 20:30:23 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Aug 31 20:30:23 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801e0260c00 x1566273973122256/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504236634 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:30:23 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Aug 31 20:30:57 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0007-osc-ffff88041b99c000: too many resent retries for object: 0:371577, rc = -11. Aug 31 20:31:03 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371577 extent [143654912-144703487] Aug 31 20:31:03 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Aug 31 20:31:03 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 305d2dcf, server f98e3eb, cksum_type 1 Aug 31 20:31:03 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Aug 31 20:31:03 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880044d02700 x1566273973294016/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504236672 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:31:03 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 6 previous similar messages Aug 31 20:31:49 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0007-osc-ffff88041b99c000: too many resent retries for object: 0:371577, rc = -11. Aug 31 20:32:25 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371577 extent [143654912-144703487] Aug 31 20:32:25 oak-gw06 kernel: LustreError: Skipped 15 previous similar messages Aug 31 20:32:25 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 305d2dcf, server f98e3eb, cksum_type 1 Aug 31 20:32:25 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 15 previous similar messages Aug 31 20:32:25 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b7128f00 x1566273973677184/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504236754 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:32:25 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 14 previous similar messages Aug 31 20:32:41 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0007-osc-ffff88041b99c000: too many resent retries for object: 0:371577, rc = -11. Aug 31 20:34:26 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0007-osc-ffff88041b99c000: too many resent retries for object: 0:371577, rc = -11. Aug 31 20:34:26 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 20:41:38 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371586 extent [174063616-175112191] Aug 31 20:41:38 oak-gw06 kernel: LustreError: Skipped 24 previous similar messages Aug 31 20:41:38 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 1ee0afad, server b1cc68a9, cksum_type 1 Aug 31 20:41:38 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 24 previous similar messages Aug 31 20:41:38 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802772b2100 x1566273975335648/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504237345 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:41:38 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 21 previous similar messages Aug 31 20:42:33 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0007-osc-ffff88041b99c000: too many resent retries for object: 0:371586, rc = -11. Aug 31 20:46:41 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371586 extent [98566144-99614719] Aug 31 20:46:41 oak-gw06 kernel: LustreError: Skipped 61 previous similar messages Aug 31 20:46:41 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client 49ee9224, server d1df3fec, cksum_type 1 Aug 31 20:46:41 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 61 previous similar messages Aug 31 20:46:41 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800a8e95b00 x1566273976069184/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504237648 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:46:41 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 56 previous similar messages Aug 31 20:56:46 oak-gw06 kernel: LustreError: 133-1: oak-OST0013-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:321756 extent [123731968-124780543] Aug 31 20:56:46 oak-gw06 kernel: LustreError: Skipped 13 previous similar messages Aug 31 20:56:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client e67bbf75, server 599ac697, cksum_type 1 Aug 31 20:56:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 13 previous similar messages Aug 31 20:56:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0013-osc-ffff88041b99c000: too many resent retries for object: 0:321756, rc = -11. Aug 31 20:56:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 5 previous similar messages Aug 31 20:56:46 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803b12c3000 x1566273977943440/t0(0) o3->oak-OST0013-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504238217 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:56:46 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 12 previous similar messages Aug 31 21:11:53 oak-gw06 kernel: LustreError: 133-1: oak-OST0013-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:321763 extent [355467264-356515839] Aug 31 21:11:53 oak-gw06 kernel: LustreError: Skipped 22 previous similar messages Aug 31 21:11:53 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client ef870ec7, server 313b9835, cksum_type 1 Aug 31 21:11:53 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 22 previous similar messages Aug 31 21:11:53 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800a2f6ea00 x1566273980167328/t0(0) o3->oak-OST0013-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504239160 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 21:11:53 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 19 previous similar messages Aug 31 21:12:48 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0013-osc-ffff88041b99c000: too many resent retries for object: 0:321763, rc = -11. Aug 31 21:12:48 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 21:21:54 oak-gw06 kernel: LustreError: 133-1: oak-OST0013-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:321753 extent [214958080-216006655] Aug 31 21:21:54 oak-gw06 kernel: LustreError: Skipped 59 previous similar messages Aug 31 21:21:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 52411125, server befad77d, cksum_type 1 Aug 31 21:21:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 59 previous similar messages Aug 31 21:21:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88038123de00 x1566273981854848/t0(0) o3->oak-OST0013-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504239736 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 21:21:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 54 previous similar messages Aug 31 22:00:30 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cd0:0x0] object 0x0:4394922 extent [2126774272-2127822847] Aug 31 22:00:30 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 22:00:30 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum 23c21036 (type 1), server csum 180dec39 (type 1), client csum now 23c21036 Aug 31 22:00:30 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 22:00:30 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88028bb20f00 x1566273991385888/t60132674342(60132674342) o4->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504242076 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 22:00:30 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 4 previous similar messages Aug 31 22:00:36 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cd0:0x0] object 0x0:4394922 extent [2126774272-2127822847] Aug 31 22:00:36 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 22:00:36 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum 23c21036 (type 1), server csum 180dec39 (type 1), client csum now 23c21036 Aug 31 22:00:36 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 22:00:45 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cd0:0x0] object 0x0:4394922 extent [2126774272-2127822847] Aug 31 22:00:45 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 31 22:00:45 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) original client csum 23c21036 (type 1), server csum 180dec39 (type 1), client csum now 23c21036 Aug 31 22:00:45 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) Skipped 1 previous similar message Aug 31 22:01:06 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cd0:0x0] object 0x0:4394922 extent [2126774272-2127822847] Aug 31 22:01:06 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 22:01:06 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum 23c21036 (type 1), server csum 180dec39 (type 1), client csum now 23c21036 Aug 31 22:01:06 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 22:01:25 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:4394922, rc = -11. Aug 31 22:01:25 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 5 previous similar messages Aug 31 22:01:53 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cb0:0x0] object 0x0:4189801 extent [4477681664-4479516671] Aug 31 22:01:53 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 22:01:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) original client csum feb62adb (type 1), server csum 1d65d4a (type 1), client csum now feb62adb Aug 31 22:01:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 22:01:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88028bba3600 x1566273991659296/t81607388970(81607388970) o4->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504242137 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 22:01:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Aug 31 22:02:48 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:4189801, rc = -11. Aug 31 22:03:15 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cd0:0x0] object 0x0:4394922 extent [2886991872-2888040447] Aug 31 22:03:15 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Aug 31 22:03:15 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum ede57b30 (type 1), server csum bdb44afd (type 1), client csum now ede57b30 Aug 31 22:03:15 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Aug 31 22:04:34 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880227bcc300 x1566273991927664/t81607390867(81607390867) o4->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 624/416 e 0 to 0 dl 1504242293 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 22:04:34 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 19 previous similar messages Aug 31 22:05:29 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cb0:0x0] object 0x0:4189801 extent [5413011456-5414322175] Aug 31 22:05:29 oak-gw06 kernel: LustreError: Skipped 20 previous similar messages Aug 31 22:05:29 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum f426da61 (type 1), server csum cf1920 (type 1), client csum now f426da61 Aug 31 22:05:29 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 20 previous similar messages Aug 31 22:05:29 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:4189801, rc = -11. Aug 31 22:05:29 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 22:09:59 oak-gw06 kernel: LustreError: 133-1: oak-OST0008-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:371498 extent [118489088-119537663] Aug 31 22:09:59 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 31 22:09:59 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client ff456f38, server d147f8b2, cksum_type 1 Aug 31 22:09:59 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Aug 31 22:09:59 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803029a4000 x1566273992868416/t0(0) o3->oak-OST0008-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504242646 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 22:09:59 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 39 previous similar messages Aug 31 22:10:54 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST0008-osc-ffff88041b99c000: too many resent retries for object: 0:371498, rc = -11. Aug 31 22:10:54 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 3 previous similar messages Aug 31 22:11:15 oak-gw06 kernel: LustreError: 133-1: oak-OST0008-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:371498 extent [118489088-119537663] Aug 31 22:11:15 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Aug 31 22:11:15 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client ff456f38, server d147f8b2, cksum_type 1 Aug 31 22:11:15 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Aug 31 22:42:23 oak-gw06 kernel: LustreError: 133-1: oak-OST0015-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4328394 extent [8388608-9437183] Aug 31 22:42:23 oak-gw06 kernel: LustreError: Skipped 15 previous similar messages Aug 31 22:42:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client a813f1e1, server d674848b, cksum_type 1 Aug 31 22:42:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 15 previous similar messages Aug 31 22:42:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802b770fc00 x1566274001083600/t0(0) o3->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504244555 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 22:42:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Aug 31 22:42:44 oak-gw06 kernel: LustreError: 133-1: oak-OST0015-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4328394 extent [8388608-9437183] Aug 31 22:42:44 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 31 22:42:44 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client a813f1e1, server d674848b, cksum_type 1 Aug 31 22:42:44 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Aug 31 22:43:18 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0015-osc-ffff88041b99c000: too many resent retries for object: 0:4328394, rc = -11. Aug 31 22:43:18 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 23:13:06 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cb0:0x0] object 0x0:4189801 extent [30926700544-30929321983] Aug 31 23:13:06 oak-gw06 kernel: LustreError: Skipped 33 previous similar messages Aug 31 23:13:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) original client csum 9f9353d3 (type 1), server csum 62151155 (type 1), client csum now 9f9353d3 Aug 31 23:13:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) Skipped 33 previous similar messages Aug 31 23:13:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802b2e1e400 x1566274011568592/t81607436328(81607436328) o4->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504246397 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 23:13:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Aug 31 23:13:21 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803bbd3c300 x1566274011636864/t81607436499(81607436499) o4->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504246412 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 23:13:21 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 4 previous similar messages Aug 31 23:13:42 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cb0:0x0] object 0x0:4189801 extent [30926700544-30929321983] Aug 31 23:13:42 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Aug 31 23:13:42 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) original client csum 9f9353d3 (type 1), server csum 62151155 (type 1), client csum now 9f9353d3 Aug 31 23:13:42 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) Skipped 7 previous similar messages Aug 31 23:13:42 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880096c4c000 x1566274011769376/t81607436661(81607436661) o4->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504246433 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 23:13:42 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 2 previous similar messages Aug 31 23:14:01 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:4189801, rc = -11. Aug 31 23:14:22 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880149c87900 x1566274012115616/t81606955923(81606955923) o4->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504246474 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 23:14:22 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 4 previous similar messages Aug 31 23:14:52 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cd6:0x0] object 0x0:4311074 extent [1254359040-1255407615] Aug 31 23:14:52 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Aug 31 23:14:52 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1355:check_write_checksum()) original client csum f3ecc038 (type 1), server csum 1e5033cc (type 1), client csum now f3ecc038 Aug 31 23:14:52 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Aug 31 23:15:11 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST0005-osc-ffff88041b99c000: too many resent retries for object: 0:4311074, rc = -11. Aug 31 23:15:29 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240059 extent [741343232-742391807] Aug 31 23:15:29 oak-gw06 kernel: LustreError: Skipped 4 previous similar messages Aug 31 23:15:29 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client a433398, server c52b2d1, cksum_type 1 Aug 31 23:15:29 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 4 previous similar messages Aug 31 23:15:35 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240059 extent [741343232-742391807] Aug 31 23:15:35 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 23:15:35 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client a433398, server c52b2d1, cksum_type 1 Aug 31 23:15:35 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Aug 31 23:15:39 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880044d00f00 x1566274012666240/t0(0) o3->oak-OST0018-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504246586 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 23:15:39 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 10 previous similar messages Aug 31 23:15:50 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240059 extent [741343232-742391807] Aug 31 23:15:50 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 23:15:50 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client a433398, server c52b2d1, cksum_type 1 Aug 31 23:15:50 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Aug 31 23:16:14 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240059 extent [741343232-742391807] Aug 31 23:16:14 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 23:16:14 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client a433398, server c52b2d1, cksum_type 1 Aug 31 23:16:14 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Aug 31 23:16:24 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST0018-osc-ffff88041b99c000: too many resent retries for object: 0:240059, rc = -11. Aug 31 23:16:52 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240059 extent [741343232-742391807] Aug 31 23:16:52 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Aug 31 23:16:52 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client a433398, server c52b2d1, cksum_type 1 Aug 31 23:16:52 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Aug 31 23:17:19 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0018-osc-ffff88041b99c000: too many resent retries for object: 0:240059, rc = -11. Aug 31 23:18:14 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240059 extent [741343232-742391807] Aug 31 23:18:14 oak-gw06 kernel: LustreError: Skipped 13 previous similar messages Aug 31 23:18:14 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client a433398, server c52b2d1, cksum_type 1 Aug 31 23:18:14 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 13 previous similar messages Aug 31 23:44:59 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240106 extent [208666624-209715199] Aug 31 23:44:59 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 5294ba90, server 13653e9a, cksum_type 1 Aug 31 23:44:59 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880275da7900 x1566274020285504/t0(0) o3->oak-OST0018-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504248310 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 23:44:59 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 25 previous similar messages Aug 31 23:45:20 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240106 extent [208666624-209715199] Aug 31 23:45:20 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 31 23:45:20 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client 5294ba90, server 13653e9a, cksum_type 1 Aug 31 23:45:20 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Aug 31 23:45:20 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802076b8300 x1566274020374048/t0(0) o3->oak-OST0018-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504248331 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 23:45:20 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Aug 31 23:45:52 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST0018-osc-ffff88041b99c000: too many resent retries for object: 0:240106, rc = -11. Aug 31 23:45:52 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 23:45:58 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240106 extent [208666624-209715199] Aug 31 23:45:58 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Aug 31 23:45:58 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 5294ba90, server 13653e9a, cksum_type 1 Aug 31 23:45:58 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Aug 31 23:45:58 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880040a98300 x1566274020471104/t0(0) o3->oak-OST0018-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504248367 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 23:45:58 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 6 previous similar messages Aug 31 23:46:44 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0018-osc-ffff88041b99c000: too many resent retries for object: 0:240106, rc = -11. Aug 31 23:47:20 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240106 extent [208666624-209715199] Aug 31 23:47:20 oak-gw06 kernel: LustreError: Skipped 15 previous similar messages Aug 31 23:47:20 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 5294ba90, server 13653e9a, cksum_type 1 Aug 31 23:47:20 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 15 previous similar messages Aug 31 23:47:20 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880112378600 x1566274020962928/t0(0) o3->oak-OST0018-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504248449 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 23:47:20 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 14 previous similar messages Aug 31 23:47:36 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0018-osc-ffff88041b99c000: too many resent retries for object: 0:240106, rc = -11. Sep 1 00:02:02 oak-gw06 kernel: LustreError: 133-1: oak-OST000f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371719 extent [121634816-122683391] Sep 1 00:02:02 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 00:02:02 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 657d44e9, server 42565e70, cksum_type 1 Sep 1 00:02:02 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Sep 1 00:02:02 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88035f979200 x1566274023561152/t0(0) o3->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504249369 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:02:02 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 1 previous similar message Sep 1 00:02:23 oak-gw06 kernel: LustreError: 133-1: oak-OST000f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371719 extent [121634816-122683391] Sep 1 00:02:23 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 00:02:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 657d44e9, server 42565e70, cksum_type 1 Sep 1 00:02:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 00:02:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800a95fea00 x1566274023573696/t0(0) o3->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504249390 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:02:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Sep 1 00:02:57 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST000f-osc-ffff88041b99c000: too many resent retries for object: 0:371719, rc = -11. Sep 1 00:26:54 oak-gw06 kernel: LustreError: 133-1: oak-OST0009-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371687 extent [2382364672-2383413247] Sep 1 00:26:54 oak-gw06 kernel: LustreError: Skipped 4 previous similar messages Sep 1 00:26:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client c29055ec, server 1f7cad1f, cksum_type 1 Sep 1 00:26:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 4 previous similar messages Sep 1 00:26:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88030969de00 x1566274030389760/t0(0) o3->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504250825 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:26:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 3 previous similar messages Sep 1 00:27:00 oak-gw06 kernel: LustreError: 133-1: oak-OST0009-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371687 extent [2382364672-2383413247] Sep 1 00:27:00 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 00:27:00 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client c29055ec, server 1f7cad1f, cksum_type 1 Sep 1 00:27:00 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Sep 1 00:27:00 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803dd3f9b00 x1566274030434416/t0(0) o3->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504250831 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:27:00 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 2 previous similar messages Sep 1 00:27:15 oak-gw06 kernel: LustreError: 133-1: oak-OST0009-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371687 extent [2382364672-2383413247] Sep 1 00:27:15 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 00:27:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client c29055ec, server 1f7cad1f, cksum_type 1 Sep 1 00:27:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Sep 1 00:27:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880038afa700 x1566274030572096/t0(0) o3->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504250846 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:27:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 2 previous similar messages Sep 1 00:27:39 oak-gw06 kernel: LustreError: 133-1: oak-OST0009-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371687 extent [2382364672-2383413247] Sep 1 00:27:39 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 00:27:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client c29055ec, server 1f7cad1f, cksum_type 1 Sep 1 00:27:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Sep 1 00:27:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802321f3000 x1566274030617184/t0(0) o3->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504250870 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:27:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 2 previous similar messages Sep 1 00:27:49 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST0009-osc-ffff88041b99c000: too many resent retries for object: 0:371687, rc = -11. Sep 1 00:28:17 oak-gw06 kernel: LustreError: 133-1: oak-OST0009-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371687 extent [2382364672-2383413247] Sep 1 00:28:17 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Sep 1 00:28:17 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client c29055ec, server 1f7cad1f, cksum_type 1 Sep 1 00:28:17 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Sep 1 00:28:17 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880028d84600 x1566274030741312/t0(0) o3->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504250908 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:28:17 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 7 previous similar messages Sep 1 00:28:44 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0009-osc-ffff88041b99c000: too many resent retries for object: 0:371687, rc = -11. Sep 1 00:29:09 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cde:0x0] object 0x0:4425060 extent [20119814144-20120862719] Sep 1 00:29:09 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 00:29:09 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum bfea48a0 (type 1), server csum b4387a93 (type 1), client csum now bfea48a0 Sep 1 00:29:09 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Sep 1 00:29:30 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cde:0x0] object 0x0:4425060 extent [20119814144-20120862719] Sep 1 00:29:30 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 00:29:30 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum bfea48a0 (type 1), server csum b4387a93 (type 1), client csum now bfea48a0 Sep 1 00:29:30 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 5 previous similar messages Sep 1 00:29:37 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880062d9db00 x1566274030976528/t64427637390(64427637390) o4->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504250988 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 00:29:37 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Sep 1 00:30:04 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cde:0x0] object 0x0:4425060 extent [20119814144-20120862719] Sep 1 00:30:04 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 00:30:04 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum bfea48a0 (type 1), server csum b4387a93 (type 1), client csum now bfea48a0 Sep 1 00:30:04 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Sep 1 00:30:04 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0009-osc-ffff88041b99c000: too many resent retries for object: 0:4425060, rc = -11. Sep 1 00:31:14 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cde:0x0] object 0x0:4425060 extent [20755251200-20757348351] Sep 1 00:31:14 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Sep 1 00:31:14 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum 18f2c218 (type 1), server csum eeb58db7 (type 1), client csum now 18f2c218 Sep 1 00:31:14 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 8 previous similar messages Sep 1 00:31:33 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0009-osc-ffff88041b99c000: too many resent retries for object: 0:4425060, rc = -11. Sep 1 00:32:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88008e424f00 x1566274031044992/t47248303493(47248303493) o4->oak-OST0024-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504251174 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 00:32:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 17 previous similar messages Sep 1 00:32:48 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0024-osc-ffff88041b99c000: too many resent retries for object: 0:2269993, rc = -11. Sep 1 00:33:39 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6ce3:0x0] object 0x0:2269993 extent [1731461120-1732509695] Sep 1 00:33:39 oak-gw06 kernel: LustreError: Skipped 13 previous similar messages Sep 1 00:33:39 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) original client csum 5f6a302f (type 1), server csum 4160f791 (type 1), client csum now 5f6a302f Sep 1 00:33:39 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) Skipped 13 previous similar messages Sep 1 00:34:31 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0024-osc-ffff88041b99c000: too many resent retries for object: 0:2269993, rc = -11. Sep 1 00:35:06 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321618 extent [221249536-222298111] Sep 1 00:35:06 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 00:35:06 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client dd8c41f7, server 3dbd43c5, cksum_type 1 Sep 1 00:35:06 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 3 previous similar messages Sep 1 00:35:58 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321618, rc = -11. Sep 1 00:36:50 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321618, rc = -11. Sep 1 00:37:11 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801c3e52a00 x1566274032094640/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504251440 ref 2 fl Interpret:RM/0/0 rc 782336/782336 Sep 1 00:37:11 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 40 previous similar messages Sep 1 00:37:42 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321618 extent [221515776-222298111] Sep 1 00:37:42 oak-gw06 kernel: LustreError: Skipped 31 previous similar messages Sep 1 00:37:42 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 4b15a28d, server ab24a0bf, cksum_type 1 Sep 1 00:37:42 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 31 previous similar messages Sep 1 00:38:37 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321618, rc = -11. Sep 1 00:38:37 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 00:47:33 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240204 extent [423624704-424673279] Sep 1 00:47:33 oak-gw06 kernel: LustreError: Skipped 22 previous similar messages Sep 1 00:47:33 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client f4e9ac32, server 93b1391f, cksum_type 1 Sep 1 00:47:33 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 22 previous similar messages Sep 1 00:47:33 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b0726700 x1566274034277536/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504252100 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:47:33 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 23 previous similar messages Sep 1 00:48:28 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:240204, rc = -11. Sep 1 00:48:28 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 00:53:58 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:240204, rc = -11. Sep 1 00:53:58 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 5 previous similar messages Sep 1 01:13:02 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240200 extent [504365056-505413631] Sep 1 01:13:02 oak-gw06 kernel: LustreError: Skipped 76 previous similar messages Sep 1 01:13:02 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 58c31509, server 656db1e4, cksum_type 1 Sep 1 01:13:02 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 76 previous similar messages Sep 1 01:13:02 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880163b23000 x1566274041407104/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504253593 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 01:13:02 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 69 previous similar messages Sep 1 01:13:57 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:240200, rc = -11. Sep 1 01:14:18 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240200 extent [504365056-505413631] Sep 1 01:14:18 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Sep 1 01:14:18 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 58c31509, server 656db1e4, cksum_type 1 Sep 1 01:14:18 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Sep 1 01:14:18 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88008d387000 x1566274041567744/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504253669 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 01:14:18 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Sep 1 01:16:50 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240200 extent [375390208-376438783] Sep 1 01:16:50 oak-gw06 kernel: LustreError: Skipped 29 previous similar messages Sep 1 01:16:50 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 736cdc82, server a9125fb1, cksum_type 1 Sep 1 01:16:50 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 29 previous similar messages Sep 1 01:16:50 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88014a3d9800 x1566274042135504/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504253859 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 01:16:50 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 26 previous similar messages Sep 1 02:12:13 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:553792 extent [6291456-7340031] Sep 1 02:12:13 oak-gw06 kernel: LustreError: Skipped 18 previous similar messages Sep 1 02:12:13 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client b89ee35, server afe48e38, cksum_type 1 Sep 1 02:12:13 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 18 previous similar messages Sep 1 02:12:13 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88026f768c00 x1566274049384432/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504257144 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 02:12:13 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 16 previous similar messages Sep 1 02:12:58 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:553792 extent [6291456-7340031] Sep 1 02:12:58 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Sep 1 02:12:58 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client b89ee35, server afe48e38, cksum_type 1 Sep 1 02:12:58 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Sep 1 02:12:58 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801fb756d00 x1566274049402032/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504257189 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 02:12:58 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 8 previous similar messages Sep 1 02:13:08 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:553792, rc = -11. Sep 1 02:13:08 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 5 previous similar messages Sep 1 02:14:13 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:553792 extent [7147520-7340031] Sep 1 02:14:13 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Sep 1 02:14:13 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 42363151, server e65b515c, cksum_type 1 Sep 1 02:14:13 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Sep 1 02:14:13 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88030664f000 x1566274049439408/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504257264 ref 2 fl Interpret:RM/0/0 rc 192512/192512 Sep 1 02:14:13 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 14 previous similar messages Sep 1 02:14:58 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:553792, rc = -11. Sep 1 02:14:58 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 02:38:12 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f83:0x154ea:0x0] object 0x0:4072530 extent [24669323264-24673255423] Sep 1 02:38:12 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Sep 1 02:38:12 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) original client csum 75275f24 (type 1), server csum 5085e4fa (type 1), client csum now 75275f24 Sep 1 02:38:12 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Sep 1 02:38:12 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88015cc71200 x1566274051064016/t51543335279(51543335279) o4->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504258739 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 02:38:12 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Sep 1 02:38:33 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802c3389500 x1566274051076416/t51543335568(51543335568) o4->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504258724 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 02:38:33 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Sep 1 02:38:48 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f83:0x154ea:0x0] object 0x0:4072530 extent [24669323264-24673255423] Sep 1 02:38:48 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Sep 1 02:38:48 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) original client csum 75275f24 (type 1), server csum 5085e4fa (type 1), client csum now 75275f24 Sep 1 02:38:48 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) Skipped 7 previous similar messages Sep 1 02:39:07 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001f-osc-ffff88041b99c000: too many resent retries for object: 0:4072530, rc = -11. Sep 1 02:39:07 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 02:39:43 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802076bbc00 x1566274051159824/t60133309618(60133309618) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 656/424 e 0 to 0 dl 1504258832 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 02:39:43 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 3 previous similar messages Sep 1 02:39:53 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6cf4:0x0] object 0x0:4468452 extent [23768334336-23770955775] Sep 1 02:39:53 oak-gw06 kernel: LustreError: Skipped 6 previous similar messages Sep 1 02:39:53 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum fb081e85 (type 1), server csum 45890a3 (type 1), client csum now fb081e85 Sep 1 02:39:53 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 6 previous similar messages Sep 1 02:40:38 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001a-osc-ffff88041b99c000: too many resent retries for object: 0:4468452, rc = -11. Sep 1 02:42:23 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cee:0x0] object 0x0:4032247 extent [2145648640-2146697215] Sep 1 02:42:23 oak-gw06 kernel: LustreError: Skipped 6 previous similar messages Sep 1 02:42:23 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum 6dbd19d5 (type 1), server csum 8611e0ec (type 1), client csum now 6dbd19d5 Sep 1 02:42:23 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 6 previous similar messages Sep 1 02:42:23 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880137f88600 x1566274051303536/t51542904695(51542904695) o4->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504258990 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 02:42:23 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Sep 1 02:43:18 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0021-osc-ffff88041b99c000: too many resent retries for object: 0:4032247, rc = -11. Sep 1 02:45:00 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88030189db00 x1566274051411600/t51543342146(51543342146) o4->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504259147 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 02:45:00 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 18 previous similar messages Sep 1 02:45:10 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST001f-osc-ffff88041b99c000: too many resent retries for object: 0:4072530, rc = -11. Sep 1 02:46:34 oak-gw06 kernel: LustreError: 133-1: oak-OST0021-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4032247 extent [715128832-716177407] Sep 1 02:46:34 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Sep 1 02:46:34 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client dbf8dbd9, server 96a89361, cksum_type 1 Sep 1 02:46:34 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Sep 1 02:46:55 oak-gw06 kernel: LustreError: 133-1: oak-OST0021-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4032247 extent [715128832-716177407] Sep 1 02:46:55 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 02:46:55 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client dbf8dbd9, server 96a89361, cksum_type 1 Sep 1 02:46:55 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 02:47:35 oak-gw06 kernel: LustreError: 133-1: oak-OST0021-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4032247 extent [715128832-716177407] Sep 1 02:47:35 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Sep 1 02:47:35 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client dbf8dbd9, server 96a89361, cksum_type 1 Sep 1 02:47:35 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Sep 1 02:48:24 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0021-osc-ffff88041b99c000: too many resent retries for object: 0:4032247, rc = -11. Sep 1 02:48:24 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 02:48:52 oak-gw06 kernel: LustreError: 133-1: oak-OST0021-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4032247 extent [715128832-716177407] Sep 1 02:48:52 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Sep 1 02:48:52 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client dbf8dbd9, server 96a89361, cksum_type 1 Sep 1 02:48:52 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 14 previous similar messages Sep 1 02:50:24 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6cf4:0x0] object 0x0:4468452 extent [27769176064-27772321791] Sep 1 02:50:24 oak-gw06 kernel: LustreError: Skipped 21 previous similar messages Sep 1 02:50:24 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1355:check_write_checksum()) original client csum 774b03a7 (type 1), server csum d4292b0c (type 1), client csum now 774b03a7 Sep 1 02:50:24 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1355:check_write_checksum()) Skipped 21 previous similar messages Sep 1 02:50:24 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88002818f000 x1566274051825840/t60133318270(60133318270) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504259435 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 02:50:24 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 30 previous similar messages Sep 1 02:54:19 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0021-osc-ffff88041b99c000: too many resent retries for object: 0:4043342, rc = -11. Sep 1 02:54:19 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 3 previous similar messages Sep 1 02:54:35 oak-gw06 kernel: LustreError: 133-1: oak-OST0021-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4043342 extent [28311552-29360127] Sep 1 02:54:35 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 02:54:35 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client b6464356, server 9c31066e, cksum_type 1 Sep 1 02:54:35 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 3 previous similar messages Sep 1 03:41:53 oak-gw06 kernel: LustreError: 133-1: oak-OST0008-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:371892 extent [41943040-42991615] Sep 1 03:41:53 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Sep 1 03:41:53 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client f14ae0e3, server a1807288, cksum_type 1 Sep 1 03:41:53 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Sep 1 03:41:53 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880122004300 x1566274058047728/t0(0) o3->oak-OST0008-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504262525 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 03:41:53 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 59 previous similar messages Sep 1 03:42:38 oak-gw06 kernel: LustreError: 133-1: oak-OST0008-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:371892 extent [41943040-42991615] Sep 1 03:42:38 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Sep 1 03:42:38 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client f14ae0e3, server a1807288, cksum_type 1 Sep 1 03:42:38 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Sep 1 03:42:48 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0008-osc-ffff88041b99c000: too many resent retries for object: 0:371892, rc = -11. Sep 1 03:42:48 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 3 previous similar messages Sep 1 03:43:09 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88020dc7a700 x1566274058115328/t0(0) o3->oak-OST0008-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504262601 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 03:43:09 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Sep 1 03:43:58 oak-gw06 kernel: LustreError: 133-1: oak-OST0008-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:371892 extent [41943040-42991615] Sep 1 03:43:58 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Sep 1 03:43:58 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client f14ae0e3, server a1807288, cksum_type 1 Sep 1 03:43:58 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Sep 1 03:44:38 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0008-osc-ffff88041b99c000: too many resent retries for object: 0:371892, rc = -11. Sep 1 03:44:38 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 03:59:28 oak-gw06 kernel: LustreError: 133-1: oak-OST0015-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322077 extent [50331648-51380223] Sep 1 03:59:28 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 03:59:28 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 9f9534b4, server 75e2b9f6, cksum_type 1 Sep 1 03:59:28 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 03:59:28 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88010164e100 x1566274059046800/t0(0) o3->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504263579 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 03:59:28 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 13 previous similar messages Sep 1 03:59:49 oak-gw06 kernel: LustreError: 133-1: oak-OST0015-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322077 extent [50331648-51380223] Sep 1 03:59:49 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 03:59:49 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 9f9534b4, server 75e2b9f6, cksum_type 1 Sep 1 03:59:49 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 03:59:49 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880297544f00 x1566274059063200/t0(0) o3->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504263600 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 03:59:49 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Sep 1 04:00:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0015-osc-ffff88041b99c000: too many resent retries for object: 0:322077, rc = -11. Sep 1 04:00:29 oak-gw06 kernel: LustreError: 133-1: oak-OST0015-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322077 extent [50331648-51380223] Sep 1 04:00:29 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Sep 1 04:00:29 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client 9f9534b4, server 75e2b9f6, cksum_type 1 Sep 1 04:00:29 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Sep 1 04:00:29 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800afd26700 x1566274059104080/t0(0) o3->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504263640 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 04:00:29 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 6 previous similar messages Sep 1 04:01:18 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0015-osc-ffff88041b99c000: too many resent retries for object: 0:322077, rc = -11. Sep 1 04:01:46 oak-gw06 kernel: LustreError: 133-1: oak-OST0015-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322077 extent [50331648-51380223] Sep 1 04:01:46 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Sep 1 04:01:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 9f9534b4, server 75e2b9f6, cksum_type 1 Sep 1 04:01:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 14 previous similar messages Sep 1 04:01:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88014b6f5b00 x1566274059173712/t0(0) o3->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504263717 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 04:01:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 13 previous similar messages Sep 1 04:02:13 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0015-osc-ffff88041b99c000: too many resent retries for object: 0:322077, rc = -11. Sep 1 04:07:57 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f83:0x154ff:0x0] object 0x0:4359803 extent [1909981184-1914961919] Sep 1 04:07:57 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Sep 1 04:07:57 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum 3cf13ab3 (type 1), server csum 8a5b1af7 (type 1), client csum now 3cf13ab3 Sep 1 04:07:57 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 32 previous similar messages Sep 1 04:07:57 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880139554c00 x1566274059452272/t60132792991(60132792991) o4->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504264089 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 04:07:57 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 2 previous similar messages Sep 1 04:08:52 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0015-osc-ffff88041b99c000: too many resent retries for object: 0:4359803, rc = -11. Sep 1 04:09:23 oak-gw06 kernel: LustreError: 133-1: oak-OST0012-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:322042 extent [37748736-38797311] Sep 1 04:09:23 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 04:09:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client 9d433ddf, server 25aeece8, cksum_type 1 Sep 1 04:09:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 3 previous similar messages Sep 1 04:12:08 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST0012-osc-ffff88041b99c000: too many resent retries for object: 0:322042, rc = -11. Sep 1 04:12:08 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 04:30:17 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d03:0x0] object 0x0:2344976 extent [18496618496-18500288511] Sep 1 04:30:17 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Sep 1 04:30:17 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) original client csum acc2f25f (type 1), server csum 64afa94e (type 1), client csum now acc2f25f Sep 1 04:30:17 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Sep 1 04:30:17 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880044d03c00 x1566274060818064/t34363704994(34363704994) o4->oak-OST0025-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504265466 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 04:30:17 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 39 previous similar messages Sep 1 04:30:27 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d03:0x0] object 0x0:2344976 extent [18496618496-18500288511] Sep 1 04:30:27 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 04:30:27 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum acc2f25f (type 1), server csum 64afa94e (type 1), client csum now acc2f25f Sep 1 04:30:27 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Sep 1 04:30:45 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d03:0x0] object 0x0:2344976 extent [18496618496-18500288511] Sep 1 04:30:45 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 04:30:45 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum acc2f25f (type 1), server csum 64afa94e (type 1), client csum now acc2f25f Sep 1 04:30:45 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Sep 1 04:31:02 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801483d3000 x1566274060859264/t34363705541(34363705541) o4->oak-OST0025-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504265511 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 04:31:02 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 8 previous similar messages Sep 1 04:31:10 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0025-osc-ffff88041b99c000: too many resent retries for object: 0:2344976, rc = -11. Sep 1 04:31:38 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d03:0x0] object 0x0:2344976 extent [18991808512-18993643519] Sep 1 04:31:38 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 04:31:38 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum ed95c992 (type 1), server csum 424e4ff1 (type 1), client csum now ed95c992 Sep 1 04:31:38 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Sep 1 04:32:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801483d3000 x1566274060931680/t34363706454(34363706454) o4->oak-OST0025-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504265590 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 04:32:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Sep 1 04:32:33 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0025-osc-ffff88041b99c000: too many resent retries for object: 0:2344976, rc = -11. Sep 1 04:32:47 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f83:0x154d0:0x0] object 0x0:4468441 extent [31287672832-31289507839] Sep 1 04:32:47 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Sep 1 04:32:47 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) original client csum a9073043 (type 1), server csum 9efde536 (type 1), client csum now a9073043 Sep 1 04:32:47 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Sep 1 04:35:02 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f83:0x154d0:0x0] object 0x0:4468441 extent [31815630848-31819825151] Sep 1 04:35:02 oak-gw06 kernel: LustreError: Skipped 20 previous similar messages Sep 1 04:35:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum 3904b96c (type 1), server csum 4f2d26f (type 1), client csum now 3904b96c Sep 1 04:35:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 20 previous similar messages Sep 1 04:35:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0018-osc-ffff88041b99c000: too many resent retries for object: 0:4468441, rc = -11. Sep 1 04:35:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 04:43:57 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f83:0x154d0:0x0] object 0x0:4468441 extent [35473063936-35474636799] Sep 1 04:43:57 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum 9fa16172 (type 1), server csum d5cff9e1 (type 1), client csum now 9fa16172 Sep 1 04:43:57 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801483d0c00 x1566274061765104/t60133891875(60133891875) o4->oak-OST0018-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504266284 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 04:43:57 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 20 previous similar messages Sep 1 04:44:52 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0018-osc-ffff88041b99c000: too many resent retries for object: 0:4468441, rc = -11. Sep 1 04:45:31 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322239 extent [5242880-6291455] Sep 1 04:45:31 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Sep 1 04:45:31 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client 93c7acf3, server bb497b77, cksum_type 1 Sep 1 04:45:31 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Sep 1 04:46:16 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322239 extent [5242880-6291455] Sep 1 04:46:16 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Sep 1 04:46:16 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 93c7acf3, server bb497b77, cksum_type 1 Sep 1 04:46:16 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Sep 1 04:47:36 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322239 extent [5242880-6291455] Sep 1 04:47:36 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Sep 1 04:47:36 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client 93c7acf3, server bb497b77, cksum_type 1 Sep 1 04:47:36 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Sep 1 04:49:01 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880218263900 x1566274062208816/t0(0) o3->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504266552 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 04:49:01 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 48 previous similar messages Sep 1 04:50:06 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0017-osc-ffff88041b99c000: too many resent retries for object: 0:322239, rc = -11. Sep 1 04:50:06 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Sep 1 04:57:44 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f83:0x154d0:0x0] object 0x0:4468441 extent [39790051328-39792672767] Sep 1 04:57:44 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Sep 1 04:57:44 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1355:check_write_checksum()) original client csum 664817f3 (type 1), server csum b5ab4f22 (type 1), client csum now 664817f3 Sep 1 04:57:44 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Sep 1 04:58:54 oak-gw06 kernel: LustreError: 133-1: oak-OST000f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:372207 extent [19922944-20971519] Sep 1 04:58:54 oak-gw06 kernel: LustreError: Skipped 27 previous similar messages Sep 1 04:58:54 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client c1dd87c4, server 3c0e1d97, cksum_type 1 Sep 1 04:58:54 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 27 previous similar messages Sep 1 04:59:04 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88025a46ea00 x1566274062729888/t0(0) o3->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504267155 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 04:59:04 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 24 previous similar messages Sep 1 05:00:44 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST000f-osc-ffff88041b99c000: too many resent retries for object: 0:372207, rc = -11. Sep 1 05:00:44 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 05:33:35 oak-gw06 kernel: LustreError: 133-1: oak-OST000f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:372265 extent [33554432-34603007] Sep 1 05:33:35 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Sep 1 05:33:35 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client dc989456, server 88e902e4, cksum_type 1 Sep 1 05:33:35 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Sep 1 05:33:35 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880134760900 x1566274066295776/t0(0) o3->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504269226 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 05:33:35 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 45 previous similar messages Sep 1 05:34:20 oak-gw06 kernel: LustreError: 133-1: oak-OST000f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:372265 extent [33554432-34603007] Sep 1 05:34:20 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Sep 1 05:34:20 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client dc989456, server 88e902e4, cksum_type 1 Sep 1 05:34:20 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Sep 1 05:34:30 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST000f-osc-ffff88041b99c000: too many resent retries for object: 0:372265, rc = -11. Sep 1 05:34:30 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 3 previous similar messages Sep 1 05:34:51 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801f62b3c00 x1566274066378720/t0(0) o3->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504269302 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 05:34:51 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Sep 1 05:35:35 oak-gw06 kernel: LustreError: 133-1: oak-OST000f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:372265 extent [33554432-34603007] Sep 1 05:35:35 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Sep 1 05:35:35 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client dc989456, server 88e902e4, cksum_type 1 Sep 1 05:35:35 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Sep 1 05:36:17 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST000f-osc-ffff88041b99c000: too many resent retries for object: 0:372265, rc = -11. Sep 1 05:36:17 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 05:37:27 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6ca6:0x0] object 0x0:4387979 extent [10193469440-10197139455] Sep 1 05:37:27 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Sep 1 05:37:27 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum 1bab083a (type 1), server csum 9dd06d77 (type 1), client csum now 1bab083a Sep 1 05:37:27 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Sep 1 05:37:27 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88030189db00 x1566274066507008/t60133410877(60133410877) o4->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 656/424 e 0 to 0 dl 1504269456 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 05:37:27 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 23 previous similar messages Sep 1 05:38:47 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f83:0x154d2:0x0] object 0x0:830081 extent [13551009792-13553631231] Sep 1 05:38:47 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Sep 1 05:38:47 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) original client csum 5142110c (type 1), server csum a7d59ee4 (type 1), client csum now 5142110c Sep 1 05:38:47 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Sep 1 05:39:42 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST002e-osc-ffff88041b99c000: too many resent retries for object: 0:830081, rc = -11. Sep 1 05:39:42 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 05:41:13 oak-gw06 kernel: LustreError: 133-1: oak-OST000a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:372182 extent [40894464-41943039] Sep 1 05:41:13 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Sep 1 05:41:13 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 5954e9e, server e427e432, cksum_type 1 Sep 1 05:41:13 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Sep 1 05:42:29 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880362a7d500 x1566274066826720/t0(0) o3->oak-OST000a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504269761 ref 2 fl Interpret:RM/0/0 rc 503808/503808 Sep 1 05:42:29 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 45 previous similar messages Sep 1 05:44:53 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST000a-osc-ffff88041b99c000: too many resent retries for object: 0:372182, rc = -11. Sep 1 05:44:53 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Sep 1 05:53:06 oak-gw06 kernel: LustreError: 133-1: oak-OST0027-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:2395485 extent [438304768-439353343] Sep 1 05:53:06 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Sep 1 05:53:06 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 21b6fc2d, server 1164d438, cksum_type 1 Sep 1 05:53:06 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Sep 1 05:53:06 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802cbdb9800 x1566274067432992/t0(0) o3->oak-OST0027-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504270434 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 05:53:06 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 33 previous similar messages Sep 1 05:54:56 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0027-osc-ffff88041b99c000: too many resent retries for object: 0:2395485, rc = -11. Sep 1 05:54:56 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 06:20:45 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d1a:0x0] object 0x0:905059 extent [1316487168-1317535743] Sep 1 06:20:45 oak-gw06 kernel: LustreError: Skipped 21 previous similar messages Sep 1 06:20:45 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum d04fcbeb (type 1), server csum 9cca509b (type 1), client csum now d04fcbeb Sep 1 06:20:45 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 21 previous similar messages Sep 1 06:20:45 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88000ad54900 x1566274071269728/t21480499916(21480499916) o4->oak-OST002f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504272057 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 06:20:45 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Sep 1 06:21:06 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d1a:0x0] object 0x0:905059 extent [1316487168-1317535743] Sep 1 06:21:06 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 06:21:06 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) original client csum d04fcbeb (type 1), server csum 9cca509b (type 1), client csum now d04fcbeb Sep 1 06:21:06 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) Skipped 5 previous similar messages Sep 1 06:21:40 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST002f-osc-ffff88041b99c000: too many resent retries for object: 0:905059, rc = -11. Sep 1 06:21:40 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 06:22:31 oak-gw06 kernel: LustreError: 133-1: oak-OST002f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:905059 extent [785383424-786431999] Sep 1 06:22:31 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Sep 1 06:22:31 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 198d09d3, server 3006e943, cksum_type 1 Sep 1 06:22:31 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Sep 1 06:22:31 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800afd27c00 x1566274071408288/t0(0) o3->oak-OST002f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504272162 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 06:22:31 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Sep 1 06:23:26 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST002f-osc-ffff88041b99c000: too many resent retries for object: 0:905059, rc = -11. Sep 1 06:23:47 oak-gw06 kernel: LustreError: 133-1: oak-OST002f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:905059 extent [785383424-786431999] Sep 1 06:23:47 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Sep 1 06:23:47 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 198d09d3, server 3006e943, cksum_type 1 Sep 1 06:23:47 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Sep 1 06:25:06 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88014d669200 x1566274071564064/t0(0) o3->oak-OST002f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504272318 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 06:25:06 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 28 previous similar messages Sep 1 06:40:27 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d1d:0x0] object 0x0:2399337 extent [1261699072-1262747647] Sep 1 06:40:27 oak-gw06 kernel: LustreError: Skipped 4 previous similar messages Sep 1 06:40:27 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum f8de5a5a (type 1), server csum 583ed300 (type 1), client csum now f8de5a5a Sep 1 06:40:27 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 4 previous similar messages Sep 1 06:40:27 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880060ed3c00 x1566274072568272/t34364223935(34364223935) o4->oak-OST0027-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504273237 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 06:40:33 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d1d:0x0] object 0x0:2399337 extent [1261699072-1262747647] Sep 1 06:40:33 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 06:40:33 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum f8de5a5a (type 1), server csum 583ed300 (type 1), client csum now f8de5a5a Sep 1 06:40:33 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Sep 1 06:40:42 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d1d:0x0] object 0x0:2399337 extent [1261699072-1262747647] Sep 1 06:40:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Sep 1 06:40:42 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum f8de5a5a (type 1), server csum 583ed300 (type 1), client csum now f8de5a5a Sep 1 06:40:42 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 1 previous similar message Sep 1 06:41:03 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d1d:0x0] object 0x0:2399337 extent [1261699072-1262747647] Sep 1 06:41:03 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 06:41:03 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum f8de5a5a (type 1), server csum 583ed300 (type 1), client csum now f8de5a5a Sep 1 06:41:03 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Sep 1 06:41:12 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88023210a700 x1566274072606144/t34364224241(34364224241) o4->oak-OST0027-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504273283 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 06:41:12 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 8 previous similar messages Sep 1 06:41:22 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST0027-osc-ffff88041b99c000: too many resent retries for object: 0:2399337, rc = -11. Sep 1 06:41:22 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 06:42:32 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d16:0x0] object 0x0:4105432 extent [18479316992-18481151999] Sep 1 06:42:32 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 06:42:32 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum 2d980cf7 (type 1), server csum 3eccabbe (type 1), client csum now 2d980cf7 Sep 1 06:42:32 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Sep 1 06:42:32 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88034b2a6a00 x1566274072676400/t51543508943(51543508943) o4->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504273363 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 06:43:27 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001f-osc-ffff88041b99c000: too many resent retries for object: 0:4105432, rc = -11. Sep 1 06:43:57 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d16:0x0] object 0x0:4105432 extent [19054460928-19055509503] Sep 1 06:43:57 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Sep 1 06:43:57 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum 15aa1380 (type 1), server csum ba8d4784 (type 1), client csum now 15aa1380 Sep 1 06:43:57 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Sep 1 06:44:52 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST001f-osc-ffff88041b99c000: too many resent retries for object: 0:4105432, rc = -11. Sep 1 06:45:18 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88030189f300 x1566274072815696/t17188960666(17188960666) o4->oak-OST002e-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504273530 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 06:45:18 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 19 previous similar messages Sep 1 06:46:14 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST002e-osc-ffff88041b99c000: too many resent retries for object: 0:830081, rc = -11. Sep 1 06:46:47 oak-gw06 kernel: LustreError: 133-1: oak-OST001c-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240954 extent [44040192-45088767] Sep 1 06:46:47 oak-gw06 kernel: LustreError: Skipped 15 previous similar messages Sep 1 06:46:47 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client 345074ed, server a22bb01f, cksum_type 1 Sep 1 06:46:47 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 15 previous similar messages Sep 1 06:47:08 oak-gw06 kernel: LustreError: 133-1: oak-OST001c-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240954 extent [44040192-45088767] Sep 1 06:47:08 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 06:47:08 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client 345074ed, server a22bb01f, cksum_type 1 Sep 1 06:47:08 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 06:47:48 oak-gw06 kernel: LustreError: 133-1: oak-OST001c-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240954 extent [44040192-45088767] Sep 1 06:47:48 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Sep 1 06:47:48 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 345074ed, server a22bb01f, cksum_type 1 Sep 1 06:47:48 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Sep 1 06:49:05 oak-gw06 kernel: LustreError: 133-1: oak-OST001c-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240954 extent [44040192-45088767] Sep 1 06:49:05 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Sep 1 06:49:05 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 345074ed, server a22bb01f, cksum_type 1 Sep 1 06:49:05 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 14 previous similar messages Sep 1 06:49:32 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST001c-osc-ffff88041b99c000: too many resent retries for object: 0:240954, rc = -11. Sep 1 06:49:32 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 06:50:27 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88029d63e400 x1566274073146544/t0(0) o3->oak-OST001c-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504273838 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 06:50:27 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 49 previous similar messages Sep 1 07:03:30 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6d19:0x0] object 0x0:4464990 extent [1492123648-1493172223] Sep 1 07:03:30 oak-gw06 kernel: LustreError: Skipped 21 previous similar messages Sep 1 07:03:30 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum 47bb6920 (type 1), server csum 130ec260 (type 1), client csum now 47bb6920 Sep 1 07:03:30 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 21 previous similar messages Sep 1 07:03:30 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88014dd22400 x1566274074780000/t64427852518(64427852518) o4->oak-OST0012-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504274621 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 07:03:30 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Sep 1 07:03:51 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6d19:0x0] object 0x0:4464990 extent [1492123648-1493172223] Sep 1 07:03:51 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 07:03:51 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum 47bb6920 (type 1), server csum 130ec260 (type 1), client csum now 47bb6920 Sep 1 07:03:51 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 5 previous similar messages Sep 1 07:04:26 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST0012-osc-ffff88041b99c000: too many resent retries for object: 0:4464990, rc = -11. Sep 1 07:04:26 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 07:04:44 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6d19:0x0] object 0x0:4464990 extent [1915486208-1917059071] Sep 1 07:04:44 oak-gw06 kernel: LustreError: Skipped 4 previous similar messages Sep 1 07:04:44 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum a78ed1bc (type 1), server csum cbefcd60 (type 1), client csum now a78ed1bc Sep 1 07:04:44 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 4 previous similar messages Sep 1 07:05:57 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4410214 extent [21417164800-21418213375] Sep 1 07:05:57 oak-gw06 kernel: LustreError: Skipped 25 previous similar messages Sep 1 07:05:57 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 8c30af7, server d9559abf, cksum_type 1 Sep 1 07:05:57 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 25 previous similar messages Sep 1 07:06:18 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4410214 extent [21417164800-21418213375] Sep 1 07:06:18 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 07:06:18 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 8c30af7, server d9559abf, cksum_type 1 Sep 1 07:06:18 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 07:06:58 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4410214 extent [21417164800-21418213375] Sep 1 07:06:58 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Sep 1 07:06:58 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 8c30af7, server d9559abf, cksum_type 1 Sep 1 07:06:58 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Sep 1 07:08:15 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4410214 extent [21417164800-21418213375] Sep 1 07:08:15 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Sep 1 07:08:15 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 8c30af7, server d9559abf, cksum_type 1 Sep 1 07:08:15 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 14 previous similar messages Sep 1 07:15:48 oak-gw06 kernel: LustreError: 133-1: oak-OST000c-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:372555 extent [19922944-20971519] Sep 1 07:15:48 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 07:15:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client b2eb8dd3, server edad80ea, cksum_type 1 Sep 1 07:15:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 3 previous similar messages Sep 1 07:15:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b35ef900 x1566274076259392/t0(0) o3->oak-OST000c-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504275397 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 07:15:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 49 previous similar messages Sep 1 07:16:43 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST000c-osc-ffff88041b99c000: too many resent retries for object: 0:372555, rc = -11. Sep 1 07:16:43 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Sep 1 07:28:07 oak-gw06 kernel: LustreError: 133-1: oak-OST0010-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:372522 extent [4194304-5242879] Sep 1 07:28:07 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Sep 1 07:28:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 79baa8e4, server de516ac0, cksum_type 1 Sep 1 07:28:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Sep 1 07:28:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b35efc00 x1566274076893792/t0(0) o3->oak-OST0010-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504276134 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 07:28:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 49 previous similar messages Sep 1 07:29:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0010-osc-ffff88041b99c000: too many resent retries for object: 0:372522, rc = -11. Sep 1 07:29:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Sep 1 07:49:12 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:554782 extent [37748736-38797311] Sep 1 07:49:12 oak-gw06 kernel: LustreError: Skipped 21 previous similar messages Sep 1 07:49:12 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client b8f2fc54, server dd8a2526, cksum_type 1 Sep 1 07:49:12 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 21 previous similar messages Sep 1 07:49:12 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88007cba3900 x1566274077333952/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504277401 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 07:49:12 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 19 previous similar messages Sep 1 07:50:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:554782, rc = -11. Sep 1 07:50:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 07:50:28 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:554782 extent [37748736-38797311] Sep 1 07:50:28 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Sep 1 07:50:28 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client b8f2fc54, server dd8a2526, cksum_type 1 Sep 1 07:50:28 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Sep 1 07:50:28 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88007cba0c00 x1566274077347696/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504277477 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 07:50:28 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Sep 1 07:51:57 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:554782, rc = -11. Sep 1 07:51:57 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 08:08:11 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322872 extent [50331648-51380223] Sep 1 08:08:11 oak-gw06 kernel: LustreError: Skipped 15 previous similar messages Sep 1 08:08:11 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client f63029b5, server af54972a, cksum_type 1 Sep 1 08:08:11 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 15 previous similar messages Sep 1 08:08:11 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802bee6ea00 x1566274077711296/t0(0) o3->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504278538 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 08:08:11 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 13 previous similar messages Sep 1 08:08:32 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322872 extent [50331648-51380223] Sep 1 08:08:32 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 08:08:32 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client f63029b5, server af54972a, cksum_type 1 Sep 1 08:08:32 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 08:08:32 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88015f082a00 x1566274077715824/t0(0) o3->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504278559 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 08:08:32 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Sep 1 08:09:06 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0017-osc-ffff88041b99c000: too many resent retries for object: 0:322872, rc = -11. Sep 1 08:09:12 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322872 extent [50331648-51380223] Sep 1 08:09:12 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Sep 1 08:09:12 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client f63029b5, server af54972a, cksum_type 1 Sep 1 08:09:12 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Sep 1 08:09:12 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b35ef600 x1566274077728992/t0(0) o3->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504278599 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 08:09:12 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 6 previous similar messages Sep 1 08:10:01 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0017-osc-ffff88041b99c000: too many resent retries for object: 0:322872, rc = -11. Sep 1 08:10:29 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322872 extent [50331648-51380223] Sep 1 08:10:29 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Sep 1 08:10:29 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client f63029b5, server af54972a, cksum_type 1 Sep 1 08:10:29 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 14 previous similar messages Sep 1 08:10:29 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802bee6c300 x1566274077743968/t0(0) o3->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504278676 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 08:10:29 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 13 previous similar messages Sep 1 08:10:56 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST0017-osc-ffff88041b99c000: too many resent retries for object: 0:322872, rc = -11. Sep 1 08:20:41 oak-gw06 kernel: bash (11172): drop_caches: 1 Sep 1 08:21:19 oak-gw06 kernel: Lustre: DEBUG MARKER: Fri Sep 1 08:21:19 2017 Sep 1 09:01:02 oak-gw06 kernel: LustreError: 133-1: oak-OST0002-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:554917 extent [47185920-48234495] Sep 1 09:01:02 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 09:01:02 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client 11a1a5b9, server dfab14ae, cksum_type 1 Sep 1 09:01:02 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 3 previous similar messages Sep 1 09:01:02 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802b3e66400 x1566274080373632/t0(0) o3->oak-OST0002-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504281713 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:01:02 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 2 previous similar messages Sep 1 09:01:23 oak-gw06 kernel: LustreError: 133-1: oak-OST0002-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:554917 extent [47185920-48234495] Sep 1 09:01:23 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 09:01:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client 11a1a5b9, server dfab14ae, cksum_type 1 Sep 1 09:01:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 09:01:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88007dadd500 x1566274080374432/t0(0) o3->oak-OST0002-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504281734 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:01:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Sep 1 09:01:57 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST0002-osc-ffff88041b99c000: too many resent retries for object: 0:554917, rc = -11. Sep 1 09:02:03 oak-gw06 kernel: LustreError: 133-1: oak-OST0002-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:554917 extent [47185920-48234495] Sep 1 09:02:03 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Sep 1 09:02:03 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 11a1a5b9, server dfab14ae, cksum_type 1 Sep 1 09:02:03 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Sep 1 09:02:03 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88007dadcc00 x1566274080401200/t0(0) o3->oak-OST0002-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504281774 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:02:03 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 6 previous similar messages Sep 1 09:02:52 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0002-osc-ffff88041b99c000: too many resent retries for object: 0:554917, rc = -11. Sep 1 09:03:20 oak-gw06 kernel: LustreError: 133-1: oak-OST0002-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:554917 extent [47185920-48234495] Sep 1 09:03:20 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Sep 1 09:03:20 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client 11a1a5b9, server dfab14ae, cksum_type 1 Sep 1 09:03:20 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 14 previous similar messages Sep 1 09:03:20 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88005bcef600 x1566274080464752/t0(0) o3->oak-OST0002-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504281851 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:03:20 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 13 previous similar messages Sep 1 09:03:47 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0002-osc-ffff88041b99c000: too many resent retries for object: 0:554917, rc = -11. Sep 1 09:04:42 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0002-osc-ffff88041b99c000: too many resent retries for object: 0:554917, rc = -11. Sep 1 09:06:10 oak-gw06 kernel: LustreError: 133-1: oak-OST001c-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:241454 extent [253755392-254803967] Sep 1 09:06:10 oak-gw06 kernel: LustreError: Skipped 25 previous similar messages Sep 1 09:06:10 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 44aec564, server 5d8a82d2, cksum_type 1 Sep 1 09:06:10 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 25 previous similar messages Sep 1 09:06:10 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88010fe69e00 x1566274080558272/t0(0) o3->oak-OST001c-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504282021 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:06:10 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 22 previous similar messages Sep 1 09:07:05 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001c-osc-ffff88041b99c000: too many resent retries for object: 0:241454, rc = -11. Sep 1 09:07:05 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 09:09:51 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST001c-osc-ffff88041b99c000: too many resent retries for object: 0:241454, rc = -11. Sep 1 09:09:51 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 09:16:50 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:241499 extent [245366784-246415359] Sep 1 09:16:50 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Sep 1 09:16:50 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 649604cc, server 52051923, cksum_type 1 Sep 1 09:16:50 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Sep 1 09:16:50 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803b5e58300 x1566274081271888/t0(0) o3->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504282659 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:16:50 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 49 previous similar messages Sep 1 09:17:45 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001a-osc-ffff88041b99c000: too many resent retries for object: 0:241499, rc = -11. Sep 1 09:17:45 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 09:26:51 oak-gw06 kernel: LustreError: 133-1: oak-OST000e-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:372990 extent [193986560-195035135] Sep 1 09:26:51 oak-gw06 kernel: LustreError: Skipped 81 previous similar messages Sep 1 09:26:51 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client b04e92f3, server 9ec63b25, cksum_type 1 Sep 1 09:26:51 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 81 previous similar messages Sep 1 09:26:51 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880314654c00 x1566274081637888/t0(0) o3->oak-OST000e-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504283258 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:26:51 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 74 previous similar messages Sep 1 09:34:09 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST000f-osc-ffff88041b99c000: too many resent retries for object: 0:373084, rc = -11. Sep 1 09:34:09 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 7 previous similar messages Sep 1 09:41:42 oak-gw06 kernel: LustreError: 133-1: oak-OST000f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:373078 extent [338690048-339738623] Sep 1 09:41:42 oak-gw06 kernel: LustreError: Skipped 38 previous similar messages Sep 1 09:41:42 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client b8369de1, server 38679a6e, cksum_type 1 Sep 1 09:41:42 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 38 previous similar messages Sep 1 09:41:42 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803800ef000 x1566274082074704/t0(0) o3->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504284149 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:41:42 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 34 previous similar messages Sep 1 09:44:27 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST000f-osc-ffff88041b99c000: too many resent retries for object: 0:373078, rc = -11. Sep 1 09:44:27 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Sep 1 10:10:33 oak-gw06 kernel: Initializing cgroup subsys cpuset