[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-1160.76.1.el7.x86_64 (mockbuild@x86-vm-41.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Jul 26 14:15:37 UTC 2022 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-1160.76.1.el7.x86_64 root=UUID=17290ef1-d9fd-4e93-8bfa-eb73d6a32647 ro console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=384M LANG=en_US.UTF-8 transparent_hugepage=madvise intel_pstate=disable intel_iommu=on iommu=pt slub_nomerge mds=full,nosmt [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffddfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffde000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000cb37fffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0xcb3800 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 00C0000000 mask FFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffde max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5860-0x000f586f] mapped at [ffffffffff200860] [ 0.000000] Base memory trampoline at [ffff8c0940099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x9c4274000, 0x9c4274fff] PGTABLE [ 0.000000] BRK [0x9c4275000, 0x9c4275fff] PGTABLE [ 0.000000] BRK [0x9c4276000, 0x9c4276fff] PGTABLE [ 0.000000] BRK [0x9c4277000, 0x9c4277fff] PGTABLE [ 0.000000] BRK [0x9c4278000, 0x9c4278fff] PGTABLE [ 0.000000] BRK [0x9c4279000, 0x9c4279fff] PGTABLE [ 0.000000] BRK [0x9c427a000, 0x9c427afff] PGTABLE [ 0.000000] RAMDISK: [mem 0x31421000-0x34a08fff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5810 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1574 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1298 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffdfb40 01758 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffdfb00 00040 [ 0.000000] ACPI: APIC 00000000bffe130c 000C0 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe13cc 00038 (v01 BOCHS BXPCHPET 00000001 BXPC 00000001) [ 0.000000] ACPI: SRAT 00000000bffe1404 00170 (v01 BOCHS BXPCSRAT 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x03 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x04 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x05 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x06 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x07 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x08 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x09 -> Node 1 [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff] [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00100000-0xbfffffff] [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0x679bfffff] [ 0.000000] SRAT: Node 1 PXM 1 [mem 0x679c00000-0xcb37fffff] [ 0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0xbfffffff] -> [mem 0x00000000-0xbfffffff] [ 0.000000] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x679bfffff] -> [mem 0x00000000-0x679bfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x679bd9000-0x679bfffff] [ 0.000000] NODE_DATA(1) allocated [mem 0xcb37d9000-0xcb37fffff] [ 0.000000] Reserving 384MB of memory at 400MB for crashkernel (System RAM: 50999MB) [ 0.000000] kvm-clock: cpu 0, msr c:b3788001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 301882779375299 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0xcb37fffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffddfff] [ 0.000000] node 0: [mem 0x100000000-0x679bfffff] [ 0.000000] node 1: [mem 0x679c00000-0xcb37fffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x679bfffff] [ 0.000000] On node 0 totalpages: 6527868 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782302 pages, LIFO batch:31 [ 0.000000] Normal zone: 89712 pages used for memmap [ 0.000000] Normal zone: 5741568 pages, LIFO batch:31 [ 0.000000] Initmem setup node 1 [mem 0x679c00000-0xcb37fffff] [ 0.000000] On node 1 totalpages: 6528000 [ 0.000000] Normal zone: 102000 pages used for memmap [ 0.000000] Normal zone: 6528000 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 10 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffde000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:10 nr_cpu_ids:10 nr_node_ids:2 [ 0.000000] percpu: Embedded 38 pages/cpu s118784 r8192 d28672 u1048576 [ 0.000000] pcpu-alloc: s118784 r8192 d28672 u1048576 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 00 01 [0] 02 03 [0] 04 -- [1] 05 06 [ 0.000000] pcpu-alloc: [1] 07 08 [1] 09 -- [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 66041c040 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 2 zonelists in Zone order, mobility grouping on. Total pages: 12851847 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-1160.76.1.el7.x86_64 root=UUID=17290ef1-d9fd-4e93-8bfa-eb73d6a32647 ro console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=384M LANG=en_US.UTF-8 transparent_hugepage=madvise intel_pstate=disable intel_iommu=on iommu=pt slub_nomerge mds=full,nosmt [ 0.000000] DMAR: IOMMU enabled [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] x86/fpu: xstate_offset[3]: 03c0, xstate_sizes[3]: 0040 [ 0.000000] x86/fpu: xstate_offset[4]: 0400, xstate_sizes[4]: 0040 [ 0.000000] x86/fpu: xstate_offset[5]: 0440, xstate_sizes[5]: 0040 [ 0.000000] x86/fpu: xstate_offset[6]: 0480, xstate_sizes[6]: 0200 [ 0.000000] x86/fpu: xstate_offset[7]: 0680, xstate_sizes[7]: 0400 [ 0.000000] x86/fpu: xstate_offset[8]: 0000, xstate_sizes[8]: 0000 [ 0.000000] x86/fpu: xstate_offset[9]: 0a80, xstate_sizes[9]: 0008 [ 0.000000] xsave: enabled xstate_bv 0x2ff, cntxt size 0xa88 using standard form [ 0.000000] Memory: 6927836k/53272576k available (7812k kernel code, 1049104k absent, 1353192k reserved, 5932k data, 1980k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=10, Nodes=2 [ 0.000000] x86/pti: Unmapping kernel while in userspace [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=10. [ 0.000000] NR_IRQS:327936 nr_irqs:504 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 418381824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2599.998 MHz processor [ 0.613998] Calibrating delay loop (skipped) preset value.. 5199.99 BogoMIPS (lpj=2599998) [ 0.616274] pid_max: default: 32768 minimum: 301 [ 0.617506] Security Framework initialized [ 0.618614] SELinux: Initializing. [ 0.620233] SELinux: Starting in permissive mode [ 0.620235] Yama: becoming mindful. [ 0.624508] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes) [ 0.638041] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes) [ 0.645115] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.646688] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.649214] Initializing cgroup subsys memory [ 0.650997] Initializing cgroup subsys devices [ 0.652215] Initializing cgroup subsys freezer [ 0.653363] Initializing cgroup subsys net_cls [ 0.654570] Initializing cgroup subsys blkio [ 0.655746] Initializing cgroup subsys perf_event [ 0.656947] Initializing cgroup subsys hugetlb [ 0.658098] Initializing cgroup subsys pids [ 0.659240] Initializing cgroup subsys net_prio [ 0.661640] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.662926] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.664227] tlb_flushall_shift: 6 [ 0.665326] FEATURE SPEC_CTRL Present [ 0.666408] FEATURE IBPB_SUPPORT Present [ 0.667492] Spectre V1 : Mitigation: Load fences, usercopy/swapgs barriers and __user pointer sanitization [ 0.669861] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.671393] Using IBRS as the default Spectre v2 mitigation for a Skylake- [ 0.672911] generation CPU. This may have a negative performance impact. [ 0.674379] Spectre V2 : Mitigation: IBRS (kernel) [ 0.675644] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp [ 0.678154] MDS: Vulnerable: Clear CPU buffers attempted, no microcode [ 0.680147] TAA: Vulnerable: Clear CPU buffers attempted, no microcode [ 0.681662] MMIO Stale Data: Vulnerable: Clear CPU buffers attempted, no microcode [ 0.686278] Freeing SMP alternatives: 28k freed [ 0.696506] ACPI: Core revision 20130517 [ 0.698140] ACPI: All ACPI Tables successfully acquired [ 0.699633] ftrace: allocating 29699 entries in 117 pages [ 0.738640] Enabling x2apic [ 0.739628] Enabled x2apic [ 0.740787] Switched APIC routing to physical x2apic. [ 0.743097] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.744564] smpboot: CPU0: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz (fam: 06, model: 55, stepping: 07) [ 0.747237] TSC deadline timer enabled [ 0.747299] Performance Events: Skylake events, Intel PMU driver. [ 0.748990] ... version: 2 [ 0.750111] ... bit width: 48 [ 0.751321] ... generic registers: 4 [ 0.752470] ... value mask: 0000ffffffffffff [ 0.753852] ... max period: 000000007fffffff [ 0.755211] ... fixed-purpose events: 3 [ 0.756361] ... event mask: 000000070000000f [ 0.757762] KVM setup paravirtual spinlock [ 0.761373] kvm-clock: cpu 1, msr c:b3788041, secondary cpu clock [ 0.763614] smpboot: CPU 1 Converting physical 0 to logical die 1 [ 0.784521] KVM setup async PF for cpu 1 [ 0.784766] kvm-clock: cpu 2, msr c:b3788081, secondary cpu clock [ 0.785961] smpboot: CPU 2 Converting physical 0 to logical die 2 [ 0.789010] kvm-stealtime: cpu 1, msr 66051c040 [ 0.806069] KVM setup async PF for cpu 2 [ 0.806326] kvm-clock: cpu 3, msr c:b37880c1, secondary cpu clock [ 0.807568] smpboot: CPU 3 Converting physical 0 to logical die 3 [ 0.810057] kvm-stealtime: cpu 2, msr 66061c040 [ 0.827631] KVM setup async PF for cpu 3 [ 0.827878] kvm-clock: cpu 4, msr c:b3788101, secondary cpu clock [ 0.829048] smpboot: CPU 4 Converting physical 0 to logical die 4 [ 0.832392] kvm-stealtime: cpu 3, msr 66071c040 [ 0.849186] KVM setup async PF for cpu 4 [ 0.760234] smpboot: Booting Node 0, Processors #1 #2 #3 #4 OK [ 0.849258] smpboot: Booting Node 1, Processors [ 0.849259] #5 [ 0.849474] kvm-clock: cpu 5, msr c:b3788141, secondary cpu clock [ 0.850739] smpboot: CPU 5 Converting physical 0 to logical die 5 [ 0.854723] kvm-stealtime: cpu 4, msr 66081c040 [ 0.870896] KVM setup async PF for cpu 5 [ 0.871168] kvm-clock: cpu 6, msr c:b3788181, secondary cpu clock [ 0.872398] smpboot: CPU 6 Converting physical 0 to logical die 6 [ 0.886656] kvm-stealtime: cpu 5, msr c99a1c040 [ 0.892495] KVM setup async PF for cpu 6 [ 0.892789] kvm-clock: cpu 7, msr c:b37881c1, secondary cpu clock [ 0.893963] smpboot: CPU 7 Converting physical 0 to logical die 7 [ 0.896542] kvm-stealtime: cpu 6, msr c99b1c040 [ 0.914137] KVM setup async PF for cpu 7 [ 0.914392] kvm-clock: cpu 8, msr c:b3788201, secondary cpu clock [ 0.915570] smpboot: CPU 8 Converting physical 0 to logical die 8 [ 0.918234] kvm-stealtime: cpu 7, msr c99c1c040 [ 0.935701] KVM setup async PF for cpu 8 [ 0.935762] #6 #7 #8 #9 OK [ 0.935979] kvm-clock: cpu 9, msr c:b3788241, secondary cpu clock [ 0.937178] smpboot: CPU 9 Converting physical 0 to logical die 9 [ 0.940504] kvm-stealtime: cpu 8, msr c99d1c040 [ 0.957291] Brought up 10 CPUs [ 0.957296] KVM setup async PF for cpu 9 [ 0.957300] kvm-stealtime: cpu 9, msr c99e1c040 [ 0.960606] smpboot: Max logical packages: 10 [ 0.961789] smpboot: Total of 10 processors activated (51999.96 BogoMIPS) [ 1.044968] node 0 initialised, 5114572 pages in 81ms [ 1.054703] node 1 initialised, 5871039 pages in 90ms [ 1.056218] devtmpfs: initialized [ 1.057229] x86/mm: Memory block size: 128MB [ 1.062344] EVM: security.selinux [ 1.063297] EVM: security.ima [ 1.064186] EVM: security.capability [ 1.066058] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 1.067487] pinctrl core: initialized pinctrl subsystem [ 1.068807] RTC time: 22:59:50, date: 12/05/22 [ 1.069996] NET: Registered protocol family 16 [ 1.071786] cpuidle: using governor haltpoll [ 1.073181] ACPI: bus type PCI registered [ 1.074214] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 1.075679] PCI: Using configuration type 1 for base access [ 1.081592] ACPI: Added _OSI(Module Device) [ 1.082829] ACPI: Added _OSI(Processor Device) [ 1.084101] ACPI: Added _OSI(3.0 _SCP Extensions) [ 1.085430] ACPI: Added _OSI(Processor Aggregator Device) [ 1.086870] ACPI: Added _OSI(Linux-Dell-Video) [ 1.088492] ACPI: EC: Look up EC in DSDT [ 1.088985] ACPI: Interpreter enabled [ 1.090133] ACPI: (supports S0 S3 S4 S5) [ 1.091322] ACPI: Using IOAPIC for interrupt routing [ 1.092711] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 1.095261] ACPI: Enabled 2 GPEs in block 00 to 0F [ 1.100436] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 1.103423] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 1.105168] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 1.106837] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 1.109992] acpiphp: Slot [3] registered [ 1.111237] acpiphp: Slot [4] registered [ 1.112455] acpiphp: Slot [5] registered [ 1.113678] acpiphp: Slot [6] registered [ 1.114897] acpiphp: Slot [7] registered [ 1.116123] acpiphp: Slot [8] registered [ 1.117328] acpiphp: Slot [9] registered [ 1.118547] acpiphp: Slot [10] registered [ 1.119768] acpiphp: Slot [11] registered [ 1.120990] acpiphp: Slot [12] registered [ 1.122217] acpiphp: Slot [13] registered [ 1.123431] acpiphp: Slot [14] registered [ 1.124659] acpiphp: Slot [15] registered [ 1.125879] acpiphp: Slot [16] registered [ 1.127099] acpiphp: Slot [17] registered [ 1.128321] acpiphp: Slot [18] registered [ 1.129552] acpiphp: Slot [19] registered [ 1.130769] acpiphp: Slot [20] registered [ 1.132556] acpiphp: Slot [21] registered [ 1.133784] acpiphp: Slot [22] registered [ 1.135029] acpiphp: Slot [23] registered [ 1.136252] acpiphp: Slot [24] registered [ 1.137475] acpiphp: Slot [25] registered [ 1.138701] acpiphp: Slot [26] registered [ 1.139917] acpiphp: Slot [27] registered [ 1.141135] acpiphp: Slot [28] registered [ 1.142358] acpiphp: Slot [29] registered [ 1.143589] acpiphp: Slot [30] registered [ 1.144806] acpiphp: Slot [31] registered [ 1.146010] PCI host bridge to bus 0000:00 [ 1.147218] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 1.148898] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 1.150588] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 1.152823] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 1.155048] pci_bus 0000:00: root bus resource [mem 0xcc0000000-0xd3fffffff window] [ 1.157339] pci_bus 0000:00: root bus resource [bus 00-ff] [ 1.158830] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 1.159172] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 1.159639] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 1.163735] pci 0000:00:01.1: reg 0x20: [io 0xc0e0-0xc0ef] [ 1.165047] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 1.166813] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 1.168434] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 1.170198] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 1.172207] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 1.172558] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 1.174746] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 1.176694] pci 0000:00:02.0: [1b36:0100] type 00 class 0x030000 [ 1.198043] pci 0000:00:02.0: reg 0x10: [mem 0xf4000000-0xf7ffffff] [ 1.207262] pci 0000:00:02.0: reg 0x14: [mem 0xf8000000-0xfbffffff] [ 1.215121] pci 0000:00:02.0: reg 0x18: [mem 0xfc050000-0xfc051fff] [ 1.223097] pci 0000:00:02.0: reg 0x1c: [io 0xc080-0xc09f] [ 1.247408] pci 0000:00:02.0: reg 0x30: [mem 0xfc040000-0xfc04ffff pref] [ 1.247797] pci 0000:00:03.0: [1af4:1004] type 00 class 0x010000 [ 1.249270] pci 0000:00:03.0: reg 0x10: [io 0xc000-0xc03f] [ 1.250585] pci 0000:00:03.0: reg 0x14: [mem 0xfc052000-0xfc052fff] [ 1.255796] pci 0000:00:03.0: reg 0x20: [mem 0xfea00000-0xfea03fff 64bit pref] [ 1.257790] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 [ 1.259097] pci 0000:00:04.0: reg 0x10: [io 0xc0a0-0xc0bf] [ 1.260262] pci 0000:00:04.0: reg 0x14: [mem 0xfc053000-0xfc053fff] [ 1.264998] pci 0000:00:04.0: reg 0x20: [mem 0xfea04000-0xfea07fff 64bit pref] [ 1.266950] pci 0000:00:04.0: reg 0x30: [mem 0xfc000000-0xfc03ffff pref] [ 1.267558] pci 0000:00:05.0: [15b3:101c] type 00 class 0x020700 [ 1.268502] pci 0000:00:05.0: reg 0x10: [mem 0xfe600000-0xfe7fffff 64bit pref] [ 1.273370] pci 0000:00:05.0: 0.000 Gb/s available PCIe bandwidth, limited by Unknown speed x0 link at 0000:00:05.0 (capable of 252.048 Gb/s with 16 GT/s x16 link) [ 1.277771] pci 0000:00:06.0: [15b3:101c] type 00 class 0x020700 [ 1.278693] pci 0000:00:06.0: reg 0x10: [mem 0xfe800000-0xfe9fffff 64bit pref] [ 1.283747] pci 0000:00:06.0: 0.000 Gb/s available PCIe bandwidth, limited by Unknown speed x0 link at 0000:00:06.0 (capable of 252.048 Gb/s with 16 GT/s x16 link) [ 1.288095] pci 0000:00:07.0: [1af4:1003] type 00 class 0x078000 [ 1.289387] pci 0000:00:07.0: reg 0x10: [io 0xc040-0xc07f] [ 1.290614] pci 0000:00:07.0: reg 0x14: [mem 0xfc054000-0xfc054fff] [ 1.295238] pci 0000:00:07.0: reg 0x20: [mem 0xfea08000-0xfea0bfff 64bit pref] [ 1.297788] pci 0000:00:08.0: [1af4:1005] type 00 class 0x00ff00 [ 1.298711] pci 0000:00:08.0: reg 0x10: [io 0xc0c0-0xc0df] [ 1.301979] pci 0000:00:08.0: reg 0x20: [mem 0xfea0c000-0xfea0ffff 64bit pref] [ 1.303672] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.305658] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.307890] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.309825] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.311726] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.314076] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 1.316419] vgaarb: loaded [ 1.317395] vgaarb: bridge control possible 0000:00:02.0 [ 1.318975] SCSI subsystem initialized [ 1.320201] ACPI: bus type USB registered [ 1.322228] usbcore: registered new interface driver usbfs [ 1.323738] usbcore: registered new interface driver hub [ 1.325237] usbcore: registered new device driver usb [ 1.326883] EDAC MC: Ver: 3.0.0 [ 1.328361] PCI: Using ACPI for IRQ routing [ 1.329615] PCI: pci_cache_line_size set to 64 bytes [ 1.329807] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.329812] e820: reserve RAM buffer [mem 0xbffde000-0xbfffffff] [ 1.329813] e820: reserve RAM buffer [mem 0xcb3800000-0xcb3ffffff] [ 1.330034] NetLabel: Initializing [ 1.331128] NetLabel: domain hash size = 128 [ 1.332425] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.333815] NetLabel: unlabeled traffic allowed by default [ 1.335471] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.337137] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.343804] amd_nb: Cannot enumerate AMD northbridges [ 1.345240] Switched to clocksource kvm-clock [ 1.350762] pnp: PnP ACPI init [ 1.352230] ACPI: bus type PNP registered [ 1.353482] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.353519] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.353552] pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.353568] pnp 00:03: [dma 2] [ 1.353577] pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.353625] pnp 00:04: Plug and Play ACPI device, IDs PNP0400 (active) [ 1.353665] pnp 00:05: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.353795] pnp: PnP ACPI: found 6 devices [ 1.355005] ACPI: bus type PNP unregistered [ 1.363100] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.363102] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.363103] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.363104] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.363105] pci_bus 0000:00: resource 8 [mem 0xcc0000000-0xd3fffffff window] [ 1.363143] NET: Registered protocol family 2 [ 1.364850] TCP established hash table entries: 524288 (order: 10, 4194304 bytes) [ 1.367592] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 1.369507] TCP: Hash tables configured (established 524288 bind 65536) [ 1.371185] TCP: reno registered [ 1.372294] UDP hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.374014] UDP-Lite hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.375875] NET: Registered protocol family 1 [ 1.377149] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.378679] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.380147] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.382311] pci 0000:00:02.0: Boot video device [ 1.382393] PCI: CLS 0 bytes, default 64 [ 1.382444] Unpacking initramfs... [ 1.802418] Freeing initrd memory: 55200k freed [ 1.816099] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.817725] software IO TLB [mem 0xbbfde000-0xbffde000] (64MB) mapped at [ffff8c09fbfde000-ffff8c09fffddfff] [ 1.821572] sha1_ssse3: Using AVX2 optimized SHA-1 implementation [ 1.823218] sha256_ssse3: Using AVX2 optimized SHA-256 implementation [ 1.825138] futex hash table entries: 4096 (order: 6, 262144 bytes) [ 1.826712] Initialise system trusted keyring [ 1.827904] audit: initializing netlink socket (disabled) [ 1.829252] type=2000 audit(1670281414.684:1): initialized [ 1.848932] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 1.850561] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 1.853681] zpool: loaded [ 1.854652] zbud: loaded [ 1.855846] VFS: Disk quotas dquot_6.5.2 [ 1.857084] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.858959] Key type big_key registered [ 1.860141] SELinux: Registering netfilter hooks [ 1.861795] NET: Registered protocol family 38 [ 1.863654] Key type asymmetric registered [ 1.864867] Asymmetric key parser 'x509' registered [ 1.866234] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) [ 1.868555] io scheduler noop registered [ 1.869739] io scheduler deadline registered (default) [ 1.871154] io scheduler cfq registered [ 1.872322] io scheduler mq-deadline registered [ 1.873636] io scheduler kyber registered [ 1.874997] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 1.876499] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 1.878175] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 1.879909] intel_idle: does not run on family 6 model 85 [ 1.879975] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 1.882164] ACPI: Power Button [PWRF] [ 1.883568] GHES: HEST is not enabled! [ 1.884771] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 1.909838] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 1.911772] Non-volatile memory driver v1.3 [ 1.924014] Linux agpgart interface v0.103 [ 1.925380] crash memory driver: version 1.1 [ 1.926695] rdac: device handler registered [ 1.927882] hp_sw: device handler registered [ 1.929093] emc: device handler registered [ 1.930412] alua: device handler registered [ 1.931719] libphy: Fixed MDIO Bus: probed [ 1.933030] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.934625] ehci-pci: EHCI PCI platform driver [ 1.935833] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 1.937297] ohci-pci: OHCI PCI platform driver [ 1.938717] uhci_hcd: USB Universal Host Controller Interface driver [ 1.940528] usbcore: registered new interface driver usbserial_generic [ 1.942305] usbserial: USB Serial support registered for generic [ 1.944022] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 1.947187] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 1.948668] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 1.950274] mousedev: PS/2 mouse device common for all mice [ 1.952771] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 1.955602] rtc_cmos 00:00: RTC can wake from S4 [ 1.956299] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 1.956614] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 1.961709] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 1.963234] rtc_cmos 00:00: alarms up to one day, y3k, 114 bytes nvram, hpet irqs [ 1.965596] hidraw: raw HID events driver (C) Jiri Kosina [ 1.966984] usbcore: registered new interface driver usbhid [ 1.968301] usbhid: USB HID core driver [ 1.969429] drop_monitor: Initializing network drop monitor service [ 1.970869] Netfilter messages via NETLINK v0.30. [ 1.972113] TCP: cubic registered [ 1.973120] Initializing XFRM netlink socket [ 1.974370] NET: Registered protocol family 10 [ 1.975838] NET: Registered protocol family 17 [ 1.976990] mpls_gso: MPLS GSO support [ 1.978259] mce: Using 10 MCE banks [ 1.979461] PM: Hibernation image not present or could not be loaded. [ 1.979464] Loading compiled-in X.509 certificates [ 1.980970] Loaded X.509 cert 'Red Hat Enterprise Linux Driver Update Program (key 3): bf57f3e87362bc7229d9f465321773dfd1f77a80' [ 1.984438] Loaded X.509 cert 'Red Hat Enterprise Linux kpatch signing key: 4d38fd864ebe18c5f0b72e3852e2014c3a676fc8' [ 1.987140] Loaded X.509 cert 'Red Hat Enterprise Linux kernel signing key: febbd8eed39cb6e49d652eb1ea1b462a3b71e432' [ 1.989564] registered taskstats version 1 [ 1.990662] page_owner is disabled [ 1.994564] Key type trusted registered [ 1.997588] Key type encrypted registered [ 1.998680] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 2.000137] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 2.002633] Magic number: 2:191:1007 [ 2.003729] memory memory46: hash matches [ 2.004963] rtc_cmos 00:00: setting system clock to 2022-12-05 22:59:51 UTC (1670281191) [ 2.007967] Freeing unused kernel memory: 1980k freed [ 2.009899] Write protecting the kernel read-only data: 12288k [ 2.012045] Freeing unused kernel memory: 368k freed [ 2.014491] Freeing unused kernel memory: 528k freed [ 2.019854] random: systemd: uninitialized urandom read (16 bytes read) [ 2.021916] random: systemd: uninitialized urandom read (16 bytes read) [ 2.023371] random: systemd: uninitialized urandom read (16 bytes read) [ 2.025907] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 2.029808] systemd[1]: Detected virtualization kvm. [ 2.031033] systemd[1]: Detected architecture x86-64. [ 2.032250] systemd[1]: Running in initial RAM disk. [ 2.034619] systemd[1]: No hostname configured. [ 2.035797] systemd[1]: Set hostname to <localhost>. [ 2.037083] random: systemd: uninitialized urandom read (16 bytes read) [ 2.038529] systemd[1]: Initializing machine ID from random generator. [ 2.073426] random: systemd: uninitialized urandom read (16 bytes read) [ 2.074906] random: systemd: uninitialized urandom read (16 bytes read) [ 2.076391] random: systemd: uninitialized urandom read (16 bytes read) [ 2.078590] random: systemd: uninitialized urandom read (16 bytes read) [ 2.080204] random: systemd: uninitialized urandom read (16 bytes read) [ 2.082024] random: systemd: uninitialized urandom read (16 bytes read) [ 2.086933] systemd[1]: Reached target Swap. [ 2.088886] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ 2.092016] systemd[1]: Reached target Local File Systems. [ 2.094302] systemd[1]: Created slice Root Slice. [ 2.096229] systemd[1]: Listening on udev Kernel Socket. [ 2.098316] systemd[1]: Listening on Journal Socket. [ 2.100279] systemd[1]: Reached target Timers. [ 2.102752] systemd[1]: Created slice System Slice. [ 2.104697] systemd[1]: Reached target Slices. [ 2.107072] systemd[1]: Starting Journal Service... [ 2.109503] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 2.113301] systemd[1]: Starting Apply Kernel Variables... [ 2.115896] systemd[1]: Starting dracut cmdline hook... [ 2.117823] systemd[1]: Reached target Paths. [ 2.119707] systemd[1]: Listening on udev Control Socket. [ 2.121797] systemd[1]: Reached target Sockets. [ 2.123977] systemd[1]: Started Journal Service. [ 2.132922] random: fast init done [ 2.241813] device-mapper: uevent: version 1.0.3 [ 2.243101] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 2.275395] RPC: Registered named UNIX socket transport module. [ 2.276803] RPC: Registered udp transport module. [ 2.278014] RPC: Registered tcp transport module. [ 2.279226] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.391111] pps_core: LinuxPPS API ver. 1 registered [ 2.393896] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> [ 2.399529] PTP clock support registered [ 2.405459] Floppy drive(s): fd0 is 2.88M AMI BIOS [ 2.408572] mlxfw: loading out-of-tree module taints kernel. [ 2.408579] mlxfw: loading out-of-tree module taints kernel. [ 2.408606] mlxfw: module verification failed: signature and/or required key missing - tainting kernel [ 2.415539] libata version 3.00 loaded. [ 2.418134] FDC 0 is a S82078B [ 2.420092] Compat-mlnx-ofed backport release: 3952e7b [ 2.421563] Backport based on mlnx_ofed/mlnx-ofa_kernel-4.0.git 3952e7b [ 2.423219] compat.git: mlnx_ofed/mlnx-ofa_kernel-4.0.git [ 2.425623] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.447816] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 2.451762] mlx5_core 0000:00:05.0: firmware version: 20.31.1014 [ 2.589688] mlx5_core 0000:00:05.0: irq 24 for MSI/MSI-X [ 2.589706] mlx5_core 0000:00:05.0: irq 25 for MSI/MSI-X [ 2.589725] mlx5_core 0000:00:05.0: irq 26 for MSI/MSI-X [ 2.589742] mlx5_core 0000:00:05.0: irq 27 for MSI/MSI-X [ 2.589762] mlx5_core 0000:00:05.0: irq 28 for MSI/MSI-X [ 2.589778] mlx5_core 0000:00:05.0: irq 29 for MSI/MSI-X [ 2.589793] mlx5_core 0000:00:05.0: irq 30 for MSI/MSI-X [ 2.589808] mlx5_core 0000:00:05.0: irq 31 for MSI/MSI-X [ 2.589826] mlx5_core 0000:00:05.0: irq 32 for MSI/MSI-X [ 2.589842] mlx5_core 0000:00:05.0: irq 33 for MSI/MSI-X [ 2.589859] mlx5_core 0000:00:05.0: irq 34 for MSI/MSI-X [ 2.604385] mlx5_core 0000:00:06.0: firmware version: 20.31.1014 [ 2.741905] mlx5_core 0000:00:06.0: irq 35 for MSI/MSI-X [ 2.741923] mlx5_core 0000:00:06.0: irq 36 for MSI/MSI-X [ 2.741937] mlx5_core 0000:00:06.0: irq 37 for MSI/MSI-X [ 2.741954] mlx5_core 0000:00:06.0: irq 38 for MSI/MSI-X [ 2.741970] mlx5_core 0000:00:06.0: irq 39 for MSI/MSI-X [ 2.741985] mlx5_core 0000:00:06.0: irq 40 for MSI/MSI-X [ 2.742000] mlx5_core 0000:00:06.0: irq 41 for MSI/MSI-X [ 2.742016] mlx5_core 0000:00:06.0: irq 42 for MSI/MSI-X [ 2.742033] mlx5_core 0000:00:06.0: irq 43 for MSI/MSI-X [ 2.742049] mlx5_core 0000:00:06.0: irq 44 for MSI/MSI-X [ 2.742063] mlx5_core 0000:00:06.0: irq 45 for MSI/MSI-X [ 2.755639] ata_piix 0000:00:01.1: version 2.13 [ 2.756180] scsi host0: ata_piix [ 2.757473] scsi host1: ata_piix [ 2.758438] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0e0 irq 14 [ 2.759850] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0e8 irq 15 [ 2.807831] virtio-pci 0000:00:03.0: irq 46 for MSI/MSI-X [ 2.807854] virtio-pci 0000:00:03.0: irq 47 for MSI/MSI-X [ 2.807879] virtio-pci 0000:00:03.0: irq 48 for MSI/MSI-X [ 2.807897] virtio-pci 0000:00:03.0: irq 49 for MSI/MSI-X [ 2.808300] virtio-pci 0000:00:07.0: irq 50 for MSI/MSI-X [ 2.808327] virtio-pci 0000:00:07.0: irq 51 for MSI/MSI-X [ 2.808545] scsi host2: Virtio SCSI HBA [ 2.811073] scsi 2:0:0:0: Direct-Access QEMU QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5 [ 2.812772] virtio-pci 0000:00:04.0: irq 52 for MSI/MSI-X [ 2.812821] virtio-pci 0000:00:04.0: irq 53 for MSI/MSI-X [ 2.812847] virtio-pci 0000:00:04.0: irq 54 for MSI/MSI-X [ 2.821280] tsc: Refined TSC clocksource calibration: 2600.103 MHz [ 2.825944] scsi 2:0:1:0: CD-ROM QEMU QEMU CD-ROM 2.5+ PQ: 0 ANSI: 5 [ 2.923823] sd 2:0:0:0: Power-on or device reset occurred [ 2.925342] sr 2:0:1:0: Power-on or device reset occurred [ 2.926749] sd 2:0:0:0: [sda] 104857600 512-byte logical blocks: (53.6 GB/50.0 GiB) [ 2.926803] sr 2:0:1:0: [sr0] scsi3-mmc drive: 16x/50x cd/rw xa/form2 cdda tray [ 2.926804] cdrom: Uniform CD-ROM driver Revision: 3.20 [ 2.926992] sr 2:0:1:0: Attached scsi CD-ROM sr0 [ 2.932354] sd 2:0:0:0: [sda] Write Protect is off [ 2.933710] sd 2:0:0:0: [sda] Mode Sense: 63 00 00 08 [ 2.933772] sd 2:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA [ 2.937477] sda: sda1 sda2 sda3 [ 2.939042] sd 2:0:0:0: [sda] Attached SCSI disk [ 3.036534] SGI XFS with ACLs, security attributes, no debug enabled [ 3.040198] XFS (sda1): Mounting V5 Filesystem [ 3.660976] XFS (sda1): Starting recovery (logdev: internal) [ 3.689874] XFS (sda1): Ending recovery (logdev: internal) [ 3.921776] random: crng init done [ 3.965093] systemd-journald[155]: Received SIGTERM from PID 1 (systemd). [ 4.033883] SELinux: Disabled at runtime. [ 4.034982] SELinux: Unregistering netfilter hooks [ 4.076250] type=1404 audit(1670281193.571:2): selinux=0 auid=4294967295 ses=4294967295 [ 4.112366] ip_tables: (C) 2000-2006 Netfilter Core Team [ 4.114984] systemd[1]: Inserted module 'ip_tables' [ 4.245270] blk_update_request: I/O error, dev fd0, sector 0 [ 4.456414] knem 1.1.4.90mlnx1: initialized [ 4.545508] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ 4.553352] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 4.556313] cryptd: max_cpu_qlen set to 1000 [ 4.558204] parport_pc 00:04: reported by Plug and Play ACPI [ 4.558260] sd 2:0:0:0: Attached scsi generic sg0 type 0 [ 4.558314] sr 2:0:1:0: Attached scsi generic sg1 type 5 [ 4.563361] parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE] [ 4.578188] AVX2 version of gcm_enc/dec engaged. [ 4.579371] AES CTR mode by8 optimization enabled [ 4.584411] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 4.587003] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 4.613466] ppdev: user-space parallel port driver [ 4.675643] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 4.677129] [drm] Device Version 0.0 [ 4.678259] [drm] Compression level 0 log level 0 [ 4.679571] [drm] 12286 io pages at offset 0x1000000 [ 4.680922] [drm] 16777216 byte draw area at offset 0x0 [ 4.682315] [drm] RAM header offset: 0x3ffe000 [ 4.696101] [TTM] Zone kernel: Available graphics memory: 25464192 kiB [ 4.697810] [TTM] Zone dma32: Available graphics memory: 2097152 kiB [ 4.699444] [TTM] Initializing pool allocator [ 4.700707] [TTM] Initializing DMA pool allocator [ 4.702034] [drm] qxl: 16M of VRAM memory size [ 4.703301] [drm] qxl: 63M of IO pages memory ready (VRAM domain) [ 4.704851] [drm] qxl: 64M of Surface memory size [ 4.709167] [drm] main mem slot 1 [f4000000,3ffe000] [ 4.710341] [drm] surface mem slot 2 [f8000000,4000000] [ 4.710682] [drm] fb mappable at 0xF4000000, size 3145728 [ 4.710682] [drm] fb: depth 24, pitch 4096, width 1024, height 768 [ 4.714661] fbcon: qxldrmfb (fb0) is primary device [ 4.727227] Console: switching to colour frame buffer device 128x48 [ 4.731883] qxl 0000:00:02.0: fb0: qxldrmfb frame buffer device [ 4.740101] [drm] Initialized qxl 0.1.0 20120117 for 0000:00:02.0 on minor 0 [ 6.269395] XFS (dm-0): Mounting V5 Filesystem [ 6.294860] XFS (dm-0): Ending clean mount [ 6.315737] systemd-journald[602]: Received request to flush runtime journal from PID 1 [ 6.359170] type=1305 audit(1670281195.853:3): audit_pid=801 old=0 auid=4294967295 ses=4294967295 res=1 [ 7.234373] blk_update_request: I/O error, dev fd0, sector 0 [ 7.269365] blk_update_request: I/O error, dev fd0, sector 0 [ 7.294367] blk_update_request: I/O error, dev fd0, sector 0 [ 7.319365] blk_update_request: I/O error, dev fd0, sector 0 [ 7.332128] ISO 9660 Extensions: Microsoft Joliet Level 3 [ 7.337045] ISO 9660 Extensions: RRIP_1991A [ 11.903079] mlx5_core 0000:00:05.0: slow_pci_heuristic:6263:(pid 1049): Max link speed = 100000, PCI BW = 0 [ 11.904277] mlx5_core 0000:00:05.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) [ 11.905255] mlx5_core 0000:00:05.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) [ 11.937399] mlx5_core 0000:00:06.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) [ 11.938368] mlx5_core 0000:00:06.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) [ 12.026989] mlx5_core 0000:00:05.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) [ 12.028256] mlx5_core 0000:00:05.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) [ 12.043845] mlx5_core 0000:00:06.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) [ 12.045051] mlx5_core 0000:00:06.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) [ 14.725759] IPv6: ADDRCONF(NETDEV_UP): ib0.8fff: link is not ready [ 14.726822] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 14.730939] IPv6: ADDRCONF(NETDEV_CHANGE): ib0.8fff: link becomes ready [ 14.742592] IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready [ 14.940785] IPv6: ADDRCONF(NETDEV_UP): ib1.8400: link is not ready [ 14.943674] IPv6: ADDRCONF(NETDEV_UP): ib1: link is not ready [ 14.947472] IPv6: ADDRCONF(NETDEV_CHANGE): ib1.8400: link becomes ready [ 14.956603] IPv6: ADDRCONF(NETDEV_CHANGE): ib1: link becomes ready [ 24.022382] TECH PREVIEW: eBPF syscall may not be fully supported. Please review provided documentation for limitations. [ 24.034403] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 24.042434] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 34.490602] Could not insert probe at lnet_parse+0: -2 [ 34.497843] Could not insert probe at lnet_parse+0: -2 [ 44.940337] Could not insert probe at lnet_parse+0: -2 [ 44.948522] Could not insert probe at lnet_parse+0: -2 [ 55.391852] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 55.400399] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 65.760494] Could not insert probe at lnet_post_send_locked+0: -2 [ 65.767641] Could not insert probe at lnet_post_send_locked+0: -2 [ 76.022031] Could not insert probe at lnet_parse+0: -2 [ 76.029673] Could not insert probe at lnet_parse+0: -2 [ 86.269681] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 86.278054] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 96.521128] Could not insert probe at lnet_post_send_locked+0: -2 [ 96.530321] Could not insert probe at lnet_post_send_locked+0: -2 [ 106.767497] Could not insert probe at lnet_send+0: -2 [ 106.776041] Could not insert probe at lnet_send+0: -2 [ 117.021987] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 117.029726] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 127.271161] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 127.278568] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 137.511340] Could not insert probe at lnet_parse+0: -2 [ 137.518267] Could not insert probe at lnet_parse+0: -2 [ 147.762606] Could not insert probe at lnet_parse+0: -2 [ 147.771027] Could not insert probe at lnet_parse+0: -2 [ 158.018438] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 158.027334] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 168.267221] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 168.274457] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 178.516115] Could not insert probe at lnet_parse+0: -2 [ 178.524636] Could not insert probe at lnet_parse+0: -2 [ 188.770484] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 188.777919] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 199.020085] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 199.028665] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 209.268320] Could not insert probe at lnet_send+0: -2 [ 209.275239] Could not insert probe at lnet_send+0: -2 [ 219.513550] Could not insert probe at lnet_parse+0: -2 [ 219.521967] Could not insert probe at lnet_parse+0: -2 [ 229.766753] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 229.775319] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 240.022182] Could not insert probe at lnet_parse+0: -2 [ 240.029382] Could not insert probe at lnet_parse+0: -2 [ 250.271416] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 250.280120] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 260.519240] Could not insert probe at lnet_parse+0: -2 [ 260.527539] Could not insert probe at lnet_parse+0: -2 [ 270.767866] Could not insert probe at lnet_parse+0: -2 [ 270.777152] Could not insert probe at lnet_parse+0: -2 [ 281.022187] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 281.030330] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 291.271727] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 291.281810] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 301.516274] Could not insert probe at lnet_parse+0: -2 [ 301.526419] Could not insert probe at lnet_parse+0: -2 [ 311.773964] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 311.783094] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 322.023879] Could not insert probe at lnet_post_send_locked+0: -2 [ 322.034178] Could not insert probe at lnet_post_send_locked+0: -2 [ 332.269435] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 332.277299] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 342.524570] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 342.533740] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 352.775919] Could not insert probe at lnet_post_send_locked+0: -2 [ 352.784140] Could not insert probe at lnet_post_send_locked+0: -2 [ 363.024078] Could not insert probe at lnet_parse+0: -2 [ 363.033547] Could not insert probe at lnet_parse+0: -2 [ 373.280222] Could not insert probe at lnet_parse+0: -2 [ 373.290144] Could not insert probe at lnet_parse+0: -2 [ 383.521365] Could not insert probe at lnet_parse+0: -2 [ 383.529193] Could not insert probe at lnet_parse+0: -2 [ 393.772825] Could not insert probe at lnet_parse+0: -2 [ 393.781768] Could not insert probe at lnet_parse+0: -2 [ 404.029469] Could not insert probe at lnet_send+0: -2 [ 404.037160] Could not insert probe at lnet_send+0: -2 [ 414.272313] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 414.281471] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 424.534041] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 424.541927] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 434.778354] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 434.788972] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 445.030713] Could not insert probe at lnet_parse+0: -2 [ 445.039805] Could not insert probe at lnet_parse+0: -2 [ 455.283516] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 455.291621] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 465.526022] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 465.535477] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 475.774830] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 475.782932] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 486.046657] Could not insert probe at lnet_parse+0: -2 [ 486.056814] Could not insert probe at lnet_parse+0: -2 [ 496.278290] Could not insert probe at lnet_parse+0: -2 [ 496.287817] Could not insert probe at lnet_parse+0: -2 [ 506.535021] Could not insert probe at lnet_parse+0: -2 [ 506.543200] Could not insert probe at lnet_parse+0: -2 [ 516.775538] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 516.785798] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 527.031112] Could not insert probe at lnet_parse+0: -2 [ 527.038917] Could not insert probe at lnet_parse+0: -2 [ 537.278367] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 537.288598] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 547.528583] Could not insert probe at lnet_parse+0: -2 [ 547.537622] Could not insert probe at lnet_parse+0: -2 [ 557.777819] Could not insert probe at lnet_send+0: -2 [ 557.786003] Could not insert probe at lnet_send+0: -2 [ 568.032161] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 568.041570] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 578.283829] Could not insert probe at lnet_parse+0: -2 [ 578.291797] Could not insert probe at lnet_parse+0: -2 [ 588.528815] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 588.538273] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 598.781923] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 598.789763] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 609.036839] Could not insert probe at lnet_send+0: -2 [ 609.046199] Could not insert probe at lnet_send+0: -2 [ 619.287941] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 619.297033] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 629.529585] Could not insert probe at lnet_parse+0: -2 [ 629.537189] Could not insert probe at lnet_parse+0: -2 [ 639.782625] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 639.790294] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 650.039304] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 650.047605] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 660.287780] Could not insert probe at lnet_parse+0: -2 [ 660.295596] Could not insert probe at lnet_parse+0: -2 [ 670.537813] Could not insert probe at lnet_post_send_locked+0: -2 [ 670.545547] Could not insert probe at lnet_post_send_locked+0: -2 [ 680.782434] Could not insert probe at lnet_send+0: -2 [ 680.789926] Could not insert probe at lnet_send+0: -2 [ 691.040882] Could not insert probe at lnet_send+0: -2 [ 691.050341] Could not insert probe at lnet_send+0: -2 [ 701.290332] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 701.298758] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 711.533244] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 711.541027] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 721.791243] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 721.800704] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 732.037983] Could not insert probe at lnet_send+0: -2 [ 732.045935] Could not insert probe at lnet_send+0: -2 [ 742.286892] Could not insert probe at lnet_parse+0: -2 [ 742.295576] Could not insert probe at lnet_parse+0: -2 [ 752.538926] Could not insert probe at lnet_send+0: -2 [ 752.549091] Could not insert probe at lnet_send+0: -2 [ 762.786127] Could not insert probe at lnet_post_send_locked+0: -2 [ 762.794098] Could not insert probe at lnet_post_send_locked+0: -2 [ 773.041696] Could not insert probe at lnet_send+0: -2 [ 773.051940] Could not insert probe at lnet_send+0: -2 [ 783.290690] Could not insert probe at lnet_parse+0: -2 [ 783.298532] Could not insert probe at lnet_parse+0: -2 [ 793.540837] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 793.549293] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 803.786540] Could not insert probe at lnet_parse+0: -2 [ 803.795656] Could not insert probe at lnet_parse+0: -2 [ 814.040992] Could not insert probe at lnet_parse+0: -2 [ 814.048961] Could not insert probe at lnet_parse+0: -2 [ 824.294394] Could not insert probe at lnet_parse+0: -2 [ 824.302433] Could not insert probe at lnet_parse+0: -2 [ 834.538007] Could not insert probe at lnet_send+0: -2 [ 834.546865] Could not insert probe at lnet_send+0: -2 [ 844.794932] Could not insert probe at lnet_parse+0: -2 [ 844.802738] Could not insert probe at lnet_parse+0: -2 [ 855.048651] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 855.057903] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 865.297802] Could not insert probe at lnet_parse+0: -2 [ 865.305716] Could not insert probe at lnet_parse+0: -2 [ 875.543577] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 875.551574] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 885.795550] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 885.804952] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 896.047747] Could not insert probe at lnet_send+0: -2 [ 896.055435] Could not insert probe at lnet_send+0: -2 [ 906.295098] Could not insert probe at lnet_parse+0: -2 [ 906.302896] Could not insert probe at lnet_parse+0: -2 [ 916.544983] Could not insert probe at lnet_parse+0: -2 [ 916.552730] Could not insert probe at lnet_parse+0: -2 [ 926.797296] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 926.805662] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 937.047909] Could not insert probe at lnet_parse+0: -2 [ 937.055749] Could not insert probe at lnet_parse+0: -2 [ 947.296405] Could not insert probe at lnet_parse+0: -2 [ 947.304317] Could not insert probe at lnet_parse+0: -2 [ 957.541824] Could not insert probe at lnet_send+0: -2 [ 957.549795] Could not insert probe at lnet_send+0: -2 [ 967.802964] Could not insert probe at lnet_send+0: -2 [ 967.812013] Could not insert probe at lnet_send+0: -2 [ 978.051271] Could not insert probe at lnet_parse+0: -2 [ 978.060717] Could not insert probe at lnet_parse+0: -2 [ 988.302442] Could not insert probe at lnet_parse+0: -2 [ 988.310730] Could not insert probe at lnet_parse+0: -2 [ 998.549354] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 998.560397] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1008.807950] Could not insert probe at lnet_parse+0: -2 [ 1008.816045] Could not insert probe at lnet_parse+0: -2 [ 1019.053661] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1019.063043] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1029.307591] Could not insert probe at lnet_parse+0: -2 [ 1029.318393] Could not insert probe at lnet_parse+0: -2 [ 1039.553448] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1039.562825] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1049.800462] Could not insert probe at lnet_post_send_locked+0: -2 [ 1049.810182] Could not insert probe at lnet_post_send_locked+0: -2 [ 1060.055955] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1060.065716] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1070.308807] Could not insert probe at lnet_parse+0: -2 [ 1070.318252] Could not insert probe at lnet_parse+0: -2 [ 1080.555516] Could not insert probe at lnet_parse+0: -2 [ 1080.564611] Could not insert probe at lnet_parse+0: -2 [ 1090.798622] Could not insert probe at lnet_send+0: -2 [ 1090.807960] Could not insert probe at lnet_send+0: -2 [ 1101.057827] Could not insert probe at lnet_parse+0: -2 [ 1101.067393] Could not insert probe at lnet_parse+0: -2 [ 1111.313388] Could not insert probe at lnet_parse+0: -2 [ 1111.322514] Could not insert probe at lnet_parse+0: -2 [ 1121.557445] Could not insert probe at lnet_send+0: -2 [ 1121.566849] Could not insert probe at lnet_send+0: -2 [ 1131.811465] Could not insert probe at lnet_post_send_locked+0: -2 [ 1131.822422] Could not insert probe at lnet_post_send_locked+0: -2 [ 1142.059431] Could not insert probe at lnet_parse+0: -2 [ 1142.068985] Could not insert probe at lnet_parse+0: -2 [ 1152.311692] Could not insert probe at lnet_send+0: -2 [ 1152.321427] Could not insert probe at lnet_send+0: -2 [ 1162.559265] Could not insert probe at lnet_send+0: -2 [ 1162.568516] Could not insert probe at lnet_send+0: -2 [ 1172.802401] Could not insert probe at lnet_post_send_locked+0: -2 [ 1172.812008] Could not insert probe at lnet_post_send_locked+0: -2 [ 1183.056874] Could not insert probe at lnet_post_send_locked+0: -2 [ 1183.064902] Could not insert probe at lnet_post_send_locked+0: -2 [ 1193.313595] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1193.323782] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1203.561947] Could not insert probe at lnet_parse+0: -2 [ 1203.571186] Could not insert probe at lnet_parse+0: -2 [ 1213.810695] Could not insert probe at lnet_parse+0: -2 [ 1213.819668] Could not insert probe at lnet_parse+0: -2 [ 1224.059560] Could not insert probe at lnet_parse+0: -2 [ 1224.069353] Could not insert probe at lnet_parse+0: -2 [ 1234.312122] Could not insert probe at lnet_parse+0: -2 [ 1234.321375] Could not insert probe at lnet_parse+0: -2 [ 1244.564684] Could not insert probe at lnet_parse+0: -2 [ 1244.573706] Could not insert probe at lnet_parse+0: -2 [ 1254.811709] Could not insert probe at lnet_post_send_locked+0: -2 [ 1254.821184] Could not insert probe at lnet_post_send_locked+0: -2 [ 1265.070722] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 1265.080775] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 1275.313710] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1275.323181] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1285.564754] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1285.573820] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1295.816545] Could not insert probe at lnet_send+0: -2 [ 1295.825570] Could not insert probe at lnet_send+0: -2 [ 1306.064838] Could not insert probe at lnet_parse+0: -2 [ 1306.074155] Could not insert probe at lnet_parse+0: -2 [ 1316.315252] Could not insert probe at lnet_parse+0: -2 [ 1316.324612] Could not insert probe at lnet_parse+0: -2 [ 1326.568630] Could not insert probe at lnet_post_send_locked+0: -2 [ 1326.578022] Could not insert probe at lnet_post_send_locked+0: -2 [ 1336.815155] Could not insert probe at lnet_send+0: -2 [ 1336.824309] Could not insert probe at lnet_send+0: -2 [ 1347.063375] Could not insert probe at lnet_parse+0: -2 [ 1347.072770] Could not insert probe at lnet_parse+0: -2 [ 1357.313363] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1357.323145] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1367.563969] Could not insert probe at lnet_post_send_locked+0: -2 [ 1367.572176] Could not insert probe at lnet_post_send_locked+0: -2 [ 1377.815404] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1377.824965] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1388.064118] Could not insert probe at lnet_parse+0: -2 [ 1388.073418] Could not insert probe at lnet_parse+0: -2 [ 1398.318550] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1398.328783] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1408.568323] Could not insert probe at lnet_send+0: -2 [ 1408.578199] Could not insert probe at lnet_send+0: -2 [ 1418.819635] Could not insert probe at lnet_parse+0: -2 [ 1418.830429] Could not insert probe at lnet_parse+0: -2 [ 1429.065011] Could not insert probe at lnet_post_send_locked+0: -2 [ 1429.074843] Could not insert probe at lnet_post_send_locked+0: -2 [ 1439.310883] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1439.319142] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1449.571062] Could not insert probe at lnet_parse+0: -2 [ 1449.580414] Could not insert probe at lnet_parse+0: -2 [ 1459.825526] Could not insert probe at lnet_parse+0: -2 [ 1459.835202] Could not insert probe at lnet_parse+0: -2 [ 1470.070272] Could not insert probe at lnet_post_send_locked+0: -2 [ 1470.079181] Could not insert probe at lnet_post_send_locked+0: -2 [ 1480.324344] Could not insert probe at lnet_send+0: -2 [ 1480.333880] Could not insert probe at lnet_send+0: -2 [ 1490.571259] Could not insert probe at lnet_post_send_locked+0: -2 [ 1490.582180] Could not insert probe at lnet_post_send_locked+0: -2 [ 1500.829295] Could not insert probe at lnet_parse+0: -2 [ 1500.838747] Could not insert probe at lnet_parse+0: -2 [ 1511.070995] Could not insert probe at lnet_send+0: -2 [ 1511.079920] Could not insert probe at lnet_send+0: -2 [ 1521.324888] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1521.334581] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1531.576214] Could not insert probe at lnet_parse+0: -2 [ 1531.585625] Could not insert probe at lnet_parse+0: -2 [ 1541.819185] Could not insert probe at lnet_parse+0: -2 [ 1541.827037] Could not insert probe at lnet_parse+0: -2 [ 1552.067389] Could not insert probe at lnet_send+0: -2 [ 1552.075241] Could not insert probe at lnet_send+0: -2 [ 1562.320205] Could not insert probe at lnet_send+0: -2 [ 1562.327870] Could not insert probe at lnet_send+0: -2 [ 1572.571185] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1572.580172] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1582.816606] Could not insert probe at lnet_parse+0: -2 [ 1582.824733] Could not insert probe at lnet_parse+0: -2 [ 1593.068242] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1593.077306] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1603.325712] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 1603.335478] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 1613.575640] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1613.584905] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1623.827424] Could not insert probe at lnet_post_send_locked+0: -2 [ 1623.838368] Could not insert probe at lnet_post_send_locked+0: -2 [ 1634.070721] Could not insert probe at lnet_send+0: -2 [ 1634.079733] Could not insert probe at lnet_send+0: -2 [ 1644.328625] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1644.338033] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1654.576743] Could not insert probe at lnet_parse+0: -2 [ 1654.584522] Could not insert probe at lnet_parse+0: -2 [ 1664.829108] Could not insert probe at lnet_parse+0: -2 [ 1664.836744] Could not insert probe at lnet_parse+0: -2 [ 1675.083964] Could not insert probe at lnet_parse+0: -2 [ 1675.093495] Could not insert probe at lnet_parse+0: -2 [ 1685.328524] Could not insert probe at lnet_parse+0: -2 [ 1685.337738] Could not insert probe at lnet_parse+0: -2 [ 1695.589285] Could not insert probe at lnet_parse+0: -2 [ 1695.599209] Could not insert probe at lnet_parse+0: -2 [ 1705.829145] Could not insert probe at lnet_parse+0: -2 [ 1705.838576] Could not insert probe at lnet_parse+0: -2 [ 1716.074781] Could not insert probe at lnet_parse+0: -2 [ 1716.083889] Could not insert probe at lnet_parse+0: -2 [ 1726.331914] Could not insert probe at lnet_send+0: -2 [ 1726.341443] Could not insert probe at lnet_send+0: -2 [ 1736.571847] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1736.579630] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1746.828716] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1746.837081] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1757.084196] Could not insert probe at lnet_parse+0: -2 [ 1757.093731] Could not insert probe at lnet_parse+0: -2 [ 1767.331077] Could not insert probe at lnet_parse+0: -2 [ 1767.341876] Could not insert probe at lnet_parse+0: -2 [ 1777.584246] Could not insert probe at lnet_parse+0: -2 [ 1777.595581] Could not insert probe at lnet_parse+0: -2 [ 1787.833543] Could not insert probe at lnet_parse+0: -2 [ 1787.842731] Could not insert probe at lnet_parse+0: -2 [ 1798.083921] Could not insert probe at lnet_send+0: -2 [ 1798.092824] Could not insert probe at lnet_send+0: -2 [ 1808.334136] Could not insert probe at lnet_parse+0: -2 [ 1808.343732] Could not insert probe at lnet_parse+0: -2 [ 1818.584102] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1818.593594] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1828.835242] Could not insert probe at lnet_parse+0: -2 [ 1828.844385] Could not insert probe at lnet_parse+0: -2 [ 1839.084067] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 1839.093608] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 1849.334786] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1849.343802] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1859.585636] Could not insert probe at lnet_parse+0: -2 [ 1859.596425] Could not insert probe at lnet_parse+0: -2 [ 1869.839161] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1869.848828] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1880.086557] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1880.096135] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 1890.340519] Could not insert probe at lnet_parse+0: -2 [ 1890.349523] Could not insert probe at lnet_parse+0: -2 [ 1900.587504] Could not insert probe at lnet_parse+0: -2 [ 1900.595067] Could not insert probe at lnet_parse+0: -2 [ 1910.842313] Could not insert probe at lnet_parse+0: -2 [ 1910.851361] Could not insert probe at lnet_parse+0: -2 [ 1921.089803] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1921.100839] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1931.340575] Could not insert probe at lnet_parse+0: -2 [ 1931.350001] Could not insert probe at lnet_parse+0: -2 [ 1941.588972] Could not insert probe at lnet_parse+0: -2 [ 1941.598495] Could not insert probe at lnet_parse+0: -2 [ 1951.835641] Could not insert probe at lnet_parse+0: -2 [ 1951.843980] Could not insert probe at lnet_parse+0: -2 [ 1962.081088] Could not insert probe at lnet_parse+0: -2 [ 1962.090638] Could not insert probe at lnet_parse+0: -2 [ 1972.335249] Could not insert probe at lnet_parse+0: -2 [ 1972.344793] Could not insert probe at lnet_parse+0: -2 [ 1982.582896] Could not insert probe at lnet_post_send_locked+0: -2 [ 1982.591009] Could not insert probe at lnet_post_send_locked+0: -2 [ 1992.846234] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 1992.855318] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2003.087629] Could not insert probe at lnet_parse+0: -2 [ 2003.097017] Could not insert probe at lnet_parse+0: -2 [ 2013.345851] Could not insert probe at lnet_parse+0: -2 [ 2013.355800] Could not insert probe at lnet_parse+0: -2 [ 2023.582848] Could not insert probe at lnet_parse+0: -2 [ 2023.593002] Could not insert probe at lnet_parse+0: -2 [ 2033.844401] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2033.854192] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2044.093418] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2044.102767] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2054.344427] Could not insert probe at lnet_post_send_locked+0: -2 [ 2054.354084] Could not insert probe at lnet_post_send_locked+0: -2 [ 2064.587596] Could not insert probe at lnet_parse+0: -2 [ 2064.597359] Could not insert probe at lnet_parse+0: -2 [ 2074.837262] Could not insert probe at lnet_post_send_locked+0: -2 [ 2074.846875] Could not insert probe at lnet_post_send_locked+0: -2 [ 2085.095925] Could not insert probe at lnet_parse+0: -2 [ 2085.105261] Could not insert probe at lnet_parse+0: -2 [ 2095.344684] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2095.354602] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2105.598974] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2105.609880] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2115.842929] Could not insert probe at lnet_parse+0: -2 [ 2115.850442] Could not insert probe at lnet_parse+0: -2 [ 2126.105783] Could not insert probe at lnet_send+0: -2 [ 2126.114865] Could not insert probe at lnet_send+0: -2 [ 2136.346809] Could not insert probe at lnet_parse+0: -2 [ 2136.356307] Could not insert probe at lnet_parse+0: -2 [ 2146.594599] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2146.602455] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2156.848431] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2156.858123] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2167.091755] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2167.101443] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2177.349500] Could not insert probe at lnet_parse+0: -2 [ 2177.359007] Could not insert probe at lnet_parse+0: -2 [ 2187.601367] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2187.611284] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2197.842381] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2197.850448] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2208.102389] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2208.112041] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2218.350776] Could not insert probe at lnet_parse+0: -2 [ 2218.360399] Could not insert probe at lnet_parse+0: -2 [ 2228.601012] Could not insert probe at lnet_parse+0: -2 [ 2228.610621] Could not insert probe at lnet_parse+0: -2 [ 2238.851572] Could not insert probe at lnet_send+0: -2 [ 2238.861419] Could not insert probe at lnet_send+0: -2 [ 2249.100205] Could not insert probe at lnet_parse+0: -2 [ 2249.109828] Could not insert probe at lnet_parse+0: -2 [ 2259.355686] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2259.364678] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2269.598200] Could not insert probe at lnet_parse+0: -2 [ 2269.607333] Could not insert probe at lnet_parse+0: -2 [ 2279.853174] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2279.862919] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2290.102142] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2290.113119] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2300.350297] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2300.359856] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2310.599928] Could not insert probe at lnet_parse+0: -2 [ 2310.608913] Could not insert probe at lnet_parse+0: -2 [ 2320.858759] Could not insert probe at lnet_send+0: -2 [ 2320.868409] Could not insert probe at lnet_send+0: -2 [ 2331.101703] Could not insert probe at lnet_send+0: -2 [ 2331.110913] Could not insert probe at lnet_send+0: -2 [ 2341.362213] Could not insert probe at lnet_parse+0: -2 [ 2341.370857] Could not insert probe at lnet_parse+0: -2 [ 2351.602066] Could not insert probe at lnet_send+0: -2 [ 2351.611346] Could not insert probe at lnet_send+0: -2 [ 2361.861860] Could not insert probe at lnet_parse+0: -2 [ 2361.870958] Could not insert probe at lnet_parse+0: -2 [ 2372.097632] Could not insert probe at lnet_send+0: -2 [ 2372.107492] Could not insert probe at lnet_send+0: -2 [ 2382.351575] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2382.359751] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2392.607483] Could not insert probe at lnet_send+0: -2 [ 2392.616980] Could not insert probe at lnet_send+0: -2 [ 2402.860190] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2402.869735] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2413.106572] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2413.115966] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2423.360184] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2423.370921] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2433.603203] Could not insert probe at lnet_parse+0: -2 [ 2433.610764] Could not insert probe at lnet_parse+0: -2 [ 2443.861171] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2443.871295] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2454.109314] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2454.117978] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2464.359275] Could not insert probe at lnet_post_send_locked+0: -2 [ 2464.369037] Could not insert probe at lnet_post_send_locked+0: -2 [ 2474.610608] Could not insert probe at lnet_post_send_locked+0: -2 [ 2474.618315] Could not insert probe at lnet_post_send_locked+0: -2 [ 2484.866294] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2484.875844] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2495.114046] Could not insert probe at lnet_parse+0: -2 [ 2495.122193] Could not insert probe at lnet_parse+0: -2 [ 2505.361298] Could not insert probe at lnet_post_send_locked+0: -2 [ 2505.370845] Could not insert probe at lnet_post_send_locked+0: -2 [ 2515.614963] Could not insert probe at lnet_parse+0: -2 [ 2515.622969] Could not insert probe at lnet_parse+0: -2 [ 2525.862037] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2525.871750] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2536.118181] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2536.127827] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2546.361591] Could not insert probe at lnet_post_send_locked+0: -2 [ 2546.369993] Could not insert probe at lnet_post_send_locked+0: -2 [ 2556.614983] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2556.623540] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2566.864399] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2566.874108] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2577.113837] Could not insert probe at lnet_post_send_locked+0: -2 [ 2577.121230] Could not insert probe at lnet_post_send_locked+0: -2 [ 2587.369891] Could not insert probe at lnet_send+0: -2 [ 2587.378545] Could not insert probe at lnet_send+0: -2 [ 2597.605599] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2597.614487] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2607.855469] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2607.865548] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2618.117440] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2618.126887] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2628.366753] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2628.376409] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2638.619116] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2638.628578] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2648.867357] Could not insert probe at lnet_parse+0: -2 [ 2648.877186] Could not insert probe at lnet_parse+0: -2 [ 2659.117264] Could not insert probe at lnet_parse+0: -2 [ 2659.126658] Could not insert probe at lnet_parse+0: -2 [ 2669.367344] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2669.376946] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2679.616179] Could not insert probe at lnet_parse+0: -2 [ 2679.625418] Could not insert probe at lnet_parse+0: -2 [ 2689.866705] Could not insert probe at lnet_parse+0: -2 [ 2689.876120] Could not insert probe at lnet_parse+0: -2 [ 2700.110000] Could not insert probe at lnet_post_send_locked+0: -2 [ 2700.119050] Could not insert probe at lnet_post_send_locked+0: -2 [ 2710.371453] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2710.382432] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2720.608116] Could not insert probe at lnet_parse+0: -2 [ 2720.618298] Could not insert probe at lnet_parse+0: -2 [ 2730.876929] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2730.886385] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2741.113731] Could not insert probe at lnet_post_send_locked+0: -2 [ 2741.121680] Could not insert probe at lnet_post_send_locked+0: -2 [ 2751.378801] Could not insert probe at lnet_parse+0: -2 [ 2751.388126] Could not insert probe at lnet_parse+0: -2 [ 2761.620599] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2761.630278] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2771.865485] Could not insert probe at lnet_parse+0: -2 [ 2771.873384] Could not insert probe at lnet_parse+0: -2 [ 2782.113524] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2782.121280] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2792.373655] Could not insert probe at lnet_parse+0: -2 [ 2792.381670] Could not insert probe at lnet_parse+0: -2 [ 2802.618562] Could not insert probe at lnet_parse+0: -2 [ 2802.627013] Could not insert probe at lnet_parse+0: -2 [ 2812.864793] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2812.874473] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2823.115300] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2823.124867] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2833.372765] Could not insert probe at lnet_parse+0: -2 [ 2833.380654] Could not insert probe at lnet_parse+0: -2 [ 2843.621781] Could not insert probe at lnet_parse+0: -2 [ 2843.632076] Could not insert probe at lnet_parse+0: -2 [ 2853.864775] Could not insert probe at lnet_post_send_locked+0: -2 [ 2853.872581] Could not insert probe at lnet_post_send_locked+0: -2 [ 2864.118615] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2864.128595] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2874.375212] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2874.384920] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2884.618020] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2884.626262] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2894.869156] Could not insert probe at lnet_parse+0: -2 [ 2894.878546] Could not insert probe at lnet_parse+0: -2 [ 2905.124135] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2905.132031] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2915.377845] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2915.386977] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2925.629870] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2925.640048] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 2935.879651] Could not insert probe at lnet_post_send_locked+0: -2 [ 2935.889142] Could not insert probe at lnet_post_send_locked+0: -2 [ 2946.123525] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2946.133518] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2956.375890] Could not insert probe at lnet_parse+0: -2 [ 2956.384067] Could not insert probe at lnet_parse+0: -2 [ 2966.622740] Could not insert probe at lnet_post_send_locked+0: -2 [ 2966.631811] Could not insert probe at lnet_post_send_locked+0: -2 [ 2976.870892] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2976.878701] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 2987.124389] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2987.134243] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 2997.379578] Could not insert probe at lnet_post_send_locked+0: -2 [ 2997.387621] Could not insert probe at lnet_post_send_locked+0: -2 [ 3007.621319] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3007.629390] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3017.873120] Could not insert probe at lnet_send+0: -2 [ 3017.881266] Could not insert probe at lnet_send+0: -2 [ 3028.124528] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3028.133841] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3038.379182] Could not insert probe at lnet_parse+0: -2 [ 3038.387194] Could not insert probe at lnet_parse+0: -2 [ 3048.624718] Could not insert probe at lnet_parse+0: -2 [ 3048.633562] Could not insert probe at lnet_parse+0: -2 [ 3058.872526] Could not insert probe at lnet_post_send_locked+0: -2 [ 3058.880430] Could not insert probe at lnet_post_send_locked+0: -2 [ 3069.125570] Could not insert probe at lnet_send+0: -2 [ 3069.135797] Could not insert probe at lnet_send+0: -2 [ 3079.386207] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3079.395942] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3089.625048] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3089.633916] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3099.874015] Could not insert probe at lnet_parse+0: -2 [ 3099.881682] Could not insert probe at lnet_parse+0: -2 [ 3110.127577] Could not insert probe at lnet_parse+0: -2 [ 3110.136969] Could not insert probe at lnet_parse+0: -2 [ 3120.383538] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3120.391359] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3130.631462] Could not insert probe at lnet_parse+0: -2 [ 3130.639818] Could not insert probe at lnet_parse+0: -2 [ 3140.880195] Could not insert probe at lnet_parse+0: -2 [ 3140.889087] Could not insert probe at lnet_parse+0: -2 [ 3151.136695] Could not insert probe at lnet_parse+0: -2 [ 3151.146568] Could not insert probe at lnet_parse+0: -2 [ 3161.383391] Could not insert probe at lnet_send+0: -2 [ 3161.391056] Could not insert probe at lnet_send+0: -2 [ 3171.631539] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3171.639333] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3181.880445] Could not insert probe at lnet_parse+0: -2 [ 3181.888187] Could not insert probe at lnet_parse+0: -2 [ 3192.138523] Could not insert probe at lnet_parse+0: -2 [ 3192.146155] Could not insert probe at lnet_parse+0: -2 [ 3202.387524] Could not insert probe at lnet_post_send_locked+0: -2 [ 3202.395612] Could not insert probe at lnet_post_send_locked+0: -2 [ 3212.633008] Could not insert probe at lnet_parse+0: -2 [ 3212.640969] Could not insert probe at lnet_parse+0: -2 [ 3222.887102] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3222.895265] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3233.134670] Could not insert probe at lnet_parse+0: -2 [ 3233.142338] Could not insert probe at lnet_parse+0: -2 [ 3243.385645] Could not insert probe at lnet_parse+0: -2 [ 3243.394571] Could not insert probe at lnet_parse+0: -2 [ 3253.642077] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3253.651764] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3263.883912] Could not insert probe at lnet_parse+0: -2 [ 3263.891724] Could not insert probe at lnet_parse+0: -2 [ 3274.139688] Could not insert probe at lnet_parse+0: -2 [ 3274.149493] Could not insert probe at lnet_parse+0: -2 [ 3284.390965] Could not insert probe at lnet_post_send_locked+0: -2 [ 3284.398706] Could not insert probe at lnet_post_send_locked+0: -2 [ 3294.637366] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3294.647029] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3304.886836] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3304.894463] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3315.142506] Could not insert probe at lnet_send+0: -2 [ 3315.151891] Could not insert probe at lnet_send+0: -2 [ 3325.391878] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3325.400307] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3335.638648] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3335.648690] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3345.895827] Could not insert probe at lnet_parse+0: -2 [ 3345.905460] Could not insert probe at lnet_parse+0: -2 [ 3356.142292] Could not insert probe at lnet_send+0: -2 [ 3356.150511] Could not insert probe at lnet_send+0: -2 [ 3366.388530] Could not insert probe at lnet_parse+0: -2 [ 3366.396782] Could not insert probe at lnet_parse+0: -2 [ 3376.637554] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3376.647195] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3386.887661] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3386.895575] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3397.142156] Could not insert probe at lnet_send+0: -2 [ 3397.151184] Could not insert probe at lnet_send+0: -2 [ 3407.392156] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3407.400175] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3417.638472] Could not insert probe at lnet_parse+0: -2 [ 3417.646819] Could not insert probe at lnet_parse+0: -2 [ 3427.889432] Could not insert probe at lnet_parse+0: -2 [ 3427.898190] Could not insert probe at lnet_parse+0: -2 [ 3438.143214] Could not insert probe at lnet_parse+0: -2 [ 3438.151143] Could not insert probe at lnet_parse+0: -2 [ 3448.394456] Could not insert probe at lnet_parse+0: -2 [ 3448.402093] Could not insert probe at lnet_parse+0: -2 [ 3458.640051] Could not insert probe at lnet_post_send_locked+0: -2 [ 3458.649511] Could not insert probe at lnet_post_send_locked+0: -2 [ 3468.899870] Could not insert probe at lnet_send+0: -2 [ 3468.908937] Could not insert probe at lnet_send+0: -2 [ 3479.141994] Could not insert probe at lnet_send+0: -2 [ 3479.150263] Could not insert probe at lnet_send+0: -2 [ 3489.400272] Could not insert probe at lnet_parse+0: -2 [ 3489.409544] Could not insert probe at lnet_parse+0: -2 [ 3499.642463] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3499.651322] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3509.897534] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3509.905471] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3520.148197] Could not insert probe at lnet_parse+0: -2 [ 3520.158278] Could not insert probe at lnet_parse+0: -2 [ 3530.400836] Could not insert probe at lnet_parse+0: -2 [ 3530.408841] Could not insert probe at lnet_parse+0: -2 [ 3540.647706] Could not insert probe at lnet_parse+0: -2 [ 3540.655626] Could not insert probe at lnet_parse+0: -2 [ 3550.896339] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3550.906029] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3561.145073] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3561.153262] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3571.400201] Could not insert probe at lnet_parse+0: -2 [ 3571.407815] Could not insert probe at lnet_parse+0: -2 [ 3581.653145] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3581.663183] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3591.898404] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3591.906589] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3602.159177] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3602.167139] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3612.396117] Could not insert probe at lnet_send+0: -2 [ 3612.404690] Could not insert probe at lnet_send+0: -2 [ 3622.643895] Could not insert probe at lnet_parse+0: -2 [ 3622.651702] Could not insert probe at lnet_parse+0: -2 [ 3632.901616] Could not insert probe at lnet_send+0: -2 [ 3632.909255] Could not insert probe at lnet_send+0: -2 [ 3643.149463] Could not insert probe at lnet_parse+0: -2 [ 3643.157199] Could not insert probe at lnet_parse+0: -2 [ 3653.399234] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3653.407121] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3663.650326] Could not insert probe at lnet_parse+0: -2 [ 3663.658212] Could not insert probe at lnet_parse+0: -2 [ 3673.914948] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3673.922920] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3684.156817] Could not insert probe at lnet_parse+0: -2 [ 3684.166339] Could not insert probe at lnet_parse+0: -2 [ 3694.399157] Could not insert probe at lnet_parse+0: -2 [ 3694.407124] Could not insert probe at lnet_parse+0: -2 [ 3704.658882] Could not insert probe at lnet_parse+0: -2 [ 3704.668097] Could not insert probe at lnet_parse+0: -2 [ 3714.906646] Could not insert probe at lnet_parse+0: -2 [ 3714.916237] Could not insert probe at lnet_parse+0: -2 [ 3725.158331] Could not insert probe at lnet_send+0: -2 [ 3725.167760] Could not insert probe at lnet_send+0: -2 [ 3735.407601] Could not insert probe at lnet_parse+0: -2 [ 3735.415294] Could not insert probe at lnet_parse+0: -2 [ 3745.652759] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3745.660409] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3755.910215] Could not insert probe at lnet_parse+0: -2 [ 3755.919826] Could not insert probe at lnet_parse+0: -2 [ 3766.160394] Could not insert probe at lnet_parse+0: -2 [ 3766.169654] Could not insert probe at lnet_parse+0: -2 [ 3776.405676] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3776.414771] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3786.663055] Could not insert probe at lnet_parse+0: -2 [ 3786.672484] Could not insert probe at lnet_parse+0: -2 [ 3796.902714] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3796.910657] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3807.154059] Could not insert probe at lnet_post_send_locked+0: -2 [ 3807.164009] Could not insert probe at lnet_post_send_locked+0: -2 [ 3817.412483] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3817.422604] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3827.659863] Could not insert probe at lnet_post_send_locked+0: -2 [ 3827.669782] Could not insert probe at lnet_post_send_locked+0: -2 [ 3837.912347] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3837.922109] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3848.164668] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3848.174547] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3858.412870] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3858.423817] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3868.667667] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3868.677381] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 3878.914565] Could not insert probe at lnet_post_send_locked+0: -2 [ 3878.923622] Could not insert probe at lnet_post_send_locked+0: -2 [ 3889.166828] Could not insert probe at lnet_parse+0: -2 [ 3889.176306] Could not insert probe at lnet_parse+0: -2 [ 3899.416445] Could not insert probe at lnet_parse+0: -2 [ 3899.425572] Could not insert probe at lnet_parse+0: -2 [ 3909.658183] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3909.667755] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3919.906538] Could not insert probe at lnet_parse+0: -2 [ 3919.916606] Could not insert probe at lnet_parse+0: -2 [ 3930.166177] Could not insert probe at lnet_parse+0: -2 [ 3930.175909] Could not insert probe at lnet_parse+0: -2 [ 3940.413986] Could not insert probe at lnet_parse+0: -2 [ 3940.423144] Could not insert probe at lnet_parse+0: -2 [ 3950.660368] Could not insert probe at lnet_parse+0: -2 [ 3950.668218] Could not insert probe at lnet_parse+0: -2 [ 3960.915536] Could not insert probe at lnet_parse+0: -2 [ 3960.924995] Could not insert probe at lnet_parse+0: -2 [ 3971.166166] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3971.175418] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 3981.409579] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3981.417552] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3991.671354] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 3991.681331] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4001.923031] Could not insert probe at lnet_send+0: -2 [ 4001.932656] Could not insert probe at lnet_send+0: -2 [ 4012.169109] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4012.178887] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4022.423742] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4022.433174] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4032.665895] Could not insert probe at lnet_parse+0: -2 [ 4032.673879] Could not insert probe at lnet_parse+0: -2 [ 4042.923215] Could not insert probe at lnet_parse+0: -2 [ 4042.931936] Could not insert probe at lnet_parse+0: -2 [ 4053.169959] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4053.179576] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4063.425174] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4063.435100] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4073.668300] Could not insert probe at lnet_post_send_locked+0: -2 [ 4073.679265] Could not insert probe at lnet_post_send_locked+0: -2 [ 4083.926211] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4083.935430] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4094.172599] Could not insert probe at lnet_parse+0: -2 [ 4094.182056] Could not insert probe at lnet_parse+0: -2 [ 4104.424583] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4104.434349] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4114.673444] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4114.682671] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4124.923551] Could not insert probe at lnet_send+0: -2 [ 4124.932913] Could not insert probe at lnet_send+0: -2 [ 4135.173537] Could not insert probe at lnet_parse+0: -2 [ 4135.182942] Could not insert probe at lnet_parse+0: -2 [ 4145.424530] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4145.434712] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4155.676182] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4155.685421] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4165.924748] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4165.933831] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4176.178230] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4176.189236] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4186.427307] Could not insert probe at lnet_parse+0: -2 [ 4186.435709] Could not insert probe at lnet_parse+0: -2 [ 4196.672191] Could not insert probe at lnet_send+0: -2 [ 4196.679968] Could not insert probe at lnet_send+0: -2 [ 4206.926747] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4206.936444] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4217.182608] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4217.191866] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4227.425119] Could not insert probe at lnet_post_send_locked+0: -2 [ 4227.433284] Could not insert probe at lnet_post_send_locked+0: -2 [ 4237.680483] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4237.688543] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4247.929774] Could not insert probe at lnet_parse+0: -2 [ 4247.938971] Could not insert probe at lnet_parse+0: -2 [ 4258.174945] Could not insert probe at lnet_post_send_locked+0: -2 [ 4258.183157] Could not insert probe at lnet_post_send_locked+0: -2 [ 4268.429602] Could not insert probe at lnet_send+0: -2 [ 4268.437932] Could not insert probe at lnet_send+0: -2 [ 4278.684319] Could not insert probe at lnet_send+0: -2 [ 4278.693161] Could not insert probe at lnet_send+0: -2 [ 4288.929552] Could not insert probe at lnet_post_send_locked+0: -2 [ 4288.937521] Could not insert probe at lnet_post_send_locked+0: -2 [ 4299.174740] Could not insert probe at lnet_parse+0: -2 [ 4299.182630] Could not insert probe at lnet_parse+0: -2 [ 4309.432725] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4309.442367] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4319.683363] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4319.692689] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4329.932008] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4329.941325] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4340.182494] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4340.192692] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4350.433661] Could not insert probe at lnet_parse+0: -2 [ 4350.443074] Could not insert probe at lnet_parse+0: -2 [ 4360.680059] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4360.689578] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4370.925793] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4370.936743] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4381.184384] Could not insert probe at lnet_parse+0: -2 [ 4381.194104] Could not insert probe at lnet_parse+0: -2 [ 4391.439555] Could not insert probe at lnet_parse+0: -2 [ 4391.448647] Could not insert probe at lnet_parse+0: -2 [ 4401.682963] Could not insert probe at lnet_parse+0: -2 [ 4401.691716] Could not insert probe at lnet_parse+0: -2 [ 4411.940300] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4411.951246] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4422.182125] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4422.192107] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4432.443372] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4432.452962] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4442.687061] Could not insert probe at lnet_parse+0: -2 [ 4442.696629] Could not insert probe at lnet_parse+0: -2 [ 4452.938802] Could not insert probe at lnet_parse+0: -2 [ 4452.948021] Could not insert probe at lnet_parse+0: -2 [ 4463.181545] Could not insert probe at lnet_send+0: -2 [ 4463.189200] Could not insert probe at lnet_send+0: -2 [ 4473.442446] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4473.452086] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4483.687332] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4483.697065] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4493.944660] Could not insert probe at lnet_parse+0: -2 [ 4493.955448] Could not insert probe at lnet_parse+0: -2 [ 4504.188072] Could not insert probe at lnet_parse+0: -2 [ 4504.197112] Could not insert probe at lnet_parse+0: -2 [ 4514.444225] Could not insert probe at lnet_send+0: -2 [ 4514.453075] Could not insert probe at lnet_send+0: -2 [ 4524.685710] Could not insert probe at lnet_parse+0: -2 [ 4524.693685] Could not insert probe at lnet_parse+0: -2 [ 4534.943791] Could not insert probe at lnet_parse+0: -2 [ 4534.953269] Could not insert probe at lnet_parse+0: -2 [ 4545.189850] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4545.199590] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4555.442424] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4555.452064] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4565.689592] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4565.699384] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4575.939432] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4575.948632] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4586.191893] Could not insert probe at lnet_parse+0: -2 [ 4586.201251] Could not insert probe at lnet_parse+0: -2 [ 4596.441746] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4596.452577] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4606.694385] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4606.704128] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4616.942194] Could not insert probe at lnet_post_send_locked+0: -2 [ 4616.951360] Could not insert probe at lnet_post_send_locked+0: -2 [ 4627.194426] Could not insert probe at lnet_post_send_locked+0: -2 [ 4627.203848] Could not insert probe at lnet_post_send_locked+0: -2 [ 4637.441155] Could not insert probe at lnet_parse+0: -2 [ 4637.450375] Could not insert probe at lnet_parse+0: -2 [ 4647.691044] Could not insert probe at lnet_parse+0: -2 [ 4647.698822] Could not insert probe at lnet_parse+0: -2 [ 4657.944878] Could not insert probe at lnet_parse+0: -2 [ 4657.954080] Could not insert probe at lnet_parse+0: -2 [ 4668.186017] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4668.194189] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4678.445050] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4678.454887] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4688.695696] Could not insert probe at lnet_send+0: -2 [ 4688.705306] Could not insert probe at lnet_send+0: -2 [ 4698.940084] Could not insert probe at lnet_post_send_locked+0: -2 [ 4698.949628] Could not insert probe at lnet_post_send_locked+0: -2 [ 4709.198655] Could not insert probe at lnet_send+0: -2 [ 4709.208139] Could not insert probe at lnet_send+0: -2 [ 4719.444718] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4719.454463] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4729.699848] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4729.709928] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4739.947060] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4739.957973] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4750.194333] Could not insert probe at lnet_parse+0: -2 [ 4750.202387] Could not insert probe at lnet_parse+0: -2 [ 4760.448155] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4760.457928] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4770.697896] Could not insert probe at lnet_parse+0: -2 [ 4770.706547] Could not insert probe at lnet_parse+0: -2 [ 4780.949636] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4780.959821] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4791.199543] Could not insert probe at lnet_parse+0: -2 [ 4791.208931] Could not insert probe at lnet_parse+0: -2 [ 4801.454946] Could not insert probe at lnet_post_send_locked+0: -2 [ 4801.462552] Could not insert probe at lnet_post_send_locked+0: -2 [ 4811.700906] Could not insert probe at lnet_post_send_locked+0: -2 [ 4811.711775] Could not insert probe at lnet_post_send_locked+0: -2 [ 4821.940660] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4821.950205] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4832.196660] Could not insert probe at lnet_parse+0: -2 [ 4832.204537] Could not insert probe at lnet_parse+0: -2 [ 4842.445897] Could not insert probe at lnet_send+0: -2 [ 4842.454008] Could not insert probe at lnet_send+0: -2 [ 4852.697943] Could not insert probe at lnet_parse+0: -2 [ 4852.705554] Could not insert probe at lnet_parse+0: -2 [ 4862.944048] Could not insert probe at lnet_post_send_locked+0: -2 [ 4862.951745] Could not insert probe at lnet_post_send_locked+0: -2 [ 4873.194878] Could not insert probe at lnet_parse+0: -2 [ 4873.203217] Could not insert probe at lnet_parse+0: -2 [ 4883.447139] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4883.455599] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4893.698716] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4893.707969] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4903.943797] Could not insert probe at lnet_send+0: -2 [ 4903.951417] Could not insert probe at lnet_send+0: -2 [ 4914.196759] Could not insert probe at lnet_parse+0: -2 [ 4914.205800] Could not insert probe at lnet_parse+0: -2 [ 4924.446974] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4924.455164] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4934.705583] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4934.713200] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4944.960961] Could not insert probe at lnet_send+0: -2 [ 4944.968879] Could not insert probe at lnet_send+0: -2 [ 4955.198593] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4955.208405] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 4965.452072] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4965.459768] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 4975.706292] Could not insert probe at lnet_send+0: -2 [ 4975.714192] Could not insert probe at lnet_send+0: -2 [ 4985.951431] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4985.960476] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 4996.201432] Could not insert probe at lnet_parse+0: -2 [ 4996.209103] Could not insert probe at lnet_parse+0: -2 [ 5006.451236] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5006.459373] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5016.705158] Could not insert probe at lnet_send+0: -2 [ 5016.712737] Could not insert probe at lnet_send+0: -2 [ 5026.950797] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5026.959041] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5037.201133] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5037.209568] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5047.458048] Could not insert probe at lnet_parse+0: -2 [ 5047.467260] Could not insert probe at lnet_parse+0: -2 [ 5057.706660] Could not insert probe at lnet_parse+0: -2 [ 5057.714376] Could not insert probe at lnet_parse+0: -2 [ 5067.954309] Could not insert probe at lnet_parse+0: -2 [ 5067.963790] Could not insert probe at lnet_parse+0: -2 [ 5078.204650] Could not insert probe at lnet_send+0: -2 [ 5078.212143] Could not insert probe at lnet_send+0: -2 [ 5088.457801] Could not insert probe at lnet_parse+0: -2 [ 5088.465562] Could not insert probe at lnet_parse+0: -2 [ 5098.708455] Could not insert probe at lnet_parse+0: -2 [ 5098.716363] Could not insert probe at lnet_parse+0: -2 [ 5108.951156] Could not insert probe at lnet_parse+0: -2 [ 5108.959339] Could not insert probe at lnet_parse+0: -2 [ 5119.204606] Could not insert probe at lnet_post_send_locked+0: -2 [ 5119.213741] Could not insert probe at lnet_post_send_locked+0: -2 [ 5129.454413] Could not insert probe at lnet_parse+0: -2 [ 5129.461996] Could not insert probe at lnet_parse+0: -2 [ 5139.709165] Could not insert probe at lnet_parse+0: -2 [ 5139.718108] Could not insert probe at lnet_parse+0: -2 [ 5149.953024] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5149.960875] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5160.205017] Could not insert probe at lnet_parse+0: -2 [ 5160.213400] Could not insert probe at lnet_parse+0: -2 [ 5170.460395] Could not insert probe at lnet_parse+0: -2 [ 5170.468799] Could not insert probe at lnet_parse+0: -2 [ 5180.713660] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5180.721823] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5190.956789] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5190.966645] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5201.207694] Could not insert probe at lnet_send+0: -2 [ 5201.215897] Could not insert probe at lnet_send+0: -2 [ 5211.459013] Could not insert probe at lnet_parse+0: -2 [ 5211.467300] Could not insert probe at lnet_parse+0: -2 [ 5221.715419] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5221.724839] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5231.960287] Could not insert probe at lnet_parse+0: -2 [ 5231.968080] Could not insert probe at lnet_parse+0: -2 [ 5242.208706] Could not insert probe at lnet_parse+0: -2 [ 5242.217493] Could not insert probe at lnet_parse+0: -2 [ 5252.458641] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5252.466576] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5262.716042] Could not insert probe at lnet_parse+0: -2 [ 5262.724392] Could not insert probe at lnet_parse+0: -2 [ 5272.963982] Could not insert probe at lnet_parse+0: -2 [ 5272.971725] Could not insert probe at lnet_parse+0: -2 [ 5283.214731] Could not insert probe at lnet_parse+0: -2 [ 5283.224059] Could not insert probe at lnet_parse+0: -2 [ 5293.461161] Could not insert probe at lnet_parse+0: -2 [ 5293.468957] Could not insert probe at lnet_parse+0: -2 [ 5303.715849] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5303.723903] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5313.960217] Could not insert probe at lnet_send+0: -2 [ 5313.969223] Could not insert probe at lnet_send+0: -2 [ 5324.212273] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5324.220868] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5334.466862] Could not insert probe at lnet_post_send_locked+0: -2 [ 5334.475062] Could not insert probe at lnet_post_send_locked+0: -2 [ 5344.721310] Could not insert probe at lnet_parse+0: -2 [ 5344.729194] Could not insert probe at lnet_parse+0: -2 [ 5354.963991] Could not insert probe at lnet_parse+0: -2 [ 5354.971860] Could not insert probe at lnet_parse+0: -2 [ 5365.219695] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5365.229209] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5375.467880] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5375.475660] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5385.715187] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5385.725064] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5395.967018] Could not insert probe at lnet_send+0: -2 [ 5395.974907] Could not insert probe at lnet_send+0: -2 [ 5406.215833] Could not insert probe at lnet_send+0: -2 [ 5406.224859] Could not insert probe at lnet_send+0: -2 [ 5416.469940] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5416.478273] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5426.719751] Could not insert probe at lnet_parse+0: -2 [ 5426.727623] Could not insert probe at lnet_parse+0: -2 [ 5436.965826] Could not insert probe at lnet_post_send_locked+0: -2 [ 5436.975046] Could not insert probe at lnet_post_send_locked+0: -2 [ 5447.216563] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5447.224404] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5457.472905] Could not insert probe at lnet_send+0: -2 [ 5457.480790] Could not insert probe at lnet_send+0: -2 [ 5467.717831] Could not insert probe at lnet_send+0: -2 [ 5467.726692] Could not insert probe at lnet_send+0: -2 [ 5477.972741] Could not insert probe at lnet_send+0: -2 [ 5477.980601] Could not insert probe at lnet_send+0: -2 [ 5488.221059] Could not insert probe at lnet_parse+0: -2 [ 5488.228993] Could not insert probe at lnet_parse+0: -2 [ 5498.477288] Could not insert probe at lnet_parse+0: -2 [ 5498.485427] Could not insert probe at lnet_parse+0: -2 [ 5508.719784] Could not insert probe at lnet_post_send_locked+0: -2 [ 5508.728819] Could not insert probe at lnet_post_send_locked+0: -2 [ 5518.970743] Could not insert probe at lnet_post_send_locked+0: -2 [ 5518.979239] Could not insert probe at lnet_post_send_locked+0: -2 [ 5529.221062] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5529.230619] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5539.469794] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5539.477950] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5549.722879] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5549.731770] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5559.971985] Could not insert probe at lnet_parse+0: -2 [ 5559.979665] Could not insert probe at lnet_parse+0: -2 [ 5570.223406] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5570.231096] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5580.474002] Could not insert probe at lnet_parse+0: -2 [ 5580.481621] Could not insert probe at lnet_parse+0: -2 [ 5590.728731] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5590.736812] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5600.973380] Could not insert probe at lnet_parse+0: -2 [ 5600.982340] Could not insert probe at lnet_parse+0: -2 [ 5611.228135] Could not insert probe at lnet_send+0: -2 [ 5611.235807] Could not insert probe at lnet_send+0: -2 [ 5621.476240] Could not insert probe at lnet_parse+0: -2 [ 5621.484184] Could not insert probe at lnet_parse+0: -2 [ 5631.723762] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5631.731477] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5641.982004] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5641.989976] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5652.229330] Could not insert probe at lnet_parse+0: -2 [ 5652.238278] Could not insert probe at lnet_parse+0: -2 [ 5662.482489] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5662.490878] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5672.723874] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5672.732180] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5682.977905] Could not insert probe at lnet_parse+0: -2 [ 5682.985534] Could not insert probe at lnet_parse+0: -2 [ 5693.235827] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5693.245722] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5703.483942] Could not insert probe at lnet_parse+0: -2 [ 5703.493342] Could not insert probe at lnet_parse+0: -2 [ 5713.729951] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5713.737624] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5723.977670] Could not insert probe at lnet_parse+0: -2 [ 5723.987125] Could not insert probe at lnet_parse+0: -2 [ 5734.228184] Could not insert probe at lnet_send+0: -2 [ 5734.236034] Could not insert probe at lnet_send+0: -2 [ 5744.485489] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5744.493824] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5754.730113] Could not insert probe at lnet_parse+0: -2 [ 5754.737678] Could not insert probe at lnet_parse+0: -2 [ 5764.983452] Could not insert probe at lnet_post_send_locked+0: -2 [ 5764.991733] Could not insert probe at lnet_post_send_locked+0: -2 [ 5775.229027] Could not insert probe at lnet_post_send_locked+0: -2 [ 5775.238242] Could not insert probe at lnet_post_send_locked+0: -2 [ 5785.486749] Could not insert probe at lnet_parse+0: -2 [ 5785.494513] Could not insert probe at lnet_parse+0: -2 [ 5795.737809] Could not insert probe at lnet_parse+0: -2 [ 5795.746944] Could not insert probe at lnet_parse+0: -2 [ 5805.980128] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5805.988058] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5816.231761] Could not insert probe at lnet_send+0: -2 [ 5816.240726] Could not insert probe at lnet_send+0: -2 [ 5826.485158] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5826.493078] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5836.732563] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5836.740384] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5846.982741] Could not insert probe at lnet_post_send_locked+0: -2 [ 5846.991784] Could not insert probe at lnet_post_send_locked+0: -2 [ 5857.236063] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5857.244166] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5867.490093] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5867.499008] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5877.739206] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5877.747012] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5887.984325] Could not insert probe at lnet_parse+0: -2 [ 5887.993490] Could not insert probe at lnet_parse+0: -2 [ 5898.234066] Could not insert probe at lnet_parse+0: -2 [ 5898.241883] Could not insert probe at lnet_parse+0: -2 [ 5908.485835] Could not insert probe at lnet_parse+0: -2 [ 5908.494542] Could not insert probe at lnet_parse+0: -2 [ 5918.747498] Could not insert probe at lnet_parse+0: -2 [ 5918.756468] Could not insert probe at lnet_parse+0: -2 [ 5928.988185] Could not insert probe at lnet_post_send_locked+0: -2 [ 5928.998337] Could not insert probe at lnet_post_send_locked+0: -2 [ 5939.237748] Could not insert probe at lnet_post_send_locked+0: -2 [ 5939.245326] Could not insert probe at lnet_post_send_locked+0: -2 [ 5949.489351] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5949.497965] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 5959.740254] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5959.750634] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 5969.989222] Could not insert probe at lnet_parse+0: -2 [ 5969.997319] Could not insert probe at lnet_parse+0: -2 [ 5980.237985] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5980.248347] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 5990.493931] Could not insert probe at lnet_parse+0: -2 [ 5990.501450] Could not insert probe at lnet_parse+0: -2 [ 6000.743110] Could not insert probe at lnet_parse+0: -2 [ 6000.750839] Could not insert probe at lnet_parse+0: -2 [ 6010.992360] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6011.002046] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6021.243698] Could not insert probe at lnet_send+0: -2 [ 6021.251650] Could not insert probe at lnet_send+0: -2 [ 6031.493242] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6031.500925] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6041.739025] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6041.747356] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6051.992467] Could not insert probe at lnet_parse+0: -2 [ 6052.001806] Could not insert probe at lnet_parse+0: -2 [ 6062.244697] Could not insert probe at lnet_send+0: -2 [ 6062.252195] Could not insert probe at lnet_send+0: -2 [ 6072.493737] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6072.501551] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6082.739476] Could not insert probe at lnet_parse+0: -2 [ 6082.748923] Could not insert probe at lnet_parse+0: -2 [ 6092.994280] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6093.002276] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6103.242447] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6103.251734] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6113.493330] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6113.501025] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6123.746046] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6123.754961] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6133.998583] Could not insert probe at lnet_parse+0: -2 [ 6134.007965] Could not insert probe at lnet_parse+0: -2 [ 6144.247956] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6144.256003] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6154.506962] Could not insert probe at lnet_send+0: -2 [ 6154.517739] Could not insert probe at lnet_send+0: -2 [ 6164.745952] Could not insert probe at lnet_parse+0: -2 [ 6164.754082] Could not insert probe at lnet_parse+0: -2 [ 6174.998689] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6175.006472] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6185.246205] Could not insert probe at lnet_send+0: -2 [ 6185.255994] Could not insert probe at lnet_send+0: -2 [ 6195.503567] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6195.511473] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6205.752724] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6205.760798] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6216.003611] Could not insert probe at lnet_send+0: -2 [ 6216.013223] Could not insert probe at lnet_send+0: -2 [ 6226.251715] Could not insert probe at lnet_send+0: -2 [ 6226.259580] Could not insert probe at lnet_send+0: -2 [ 6236.501869] Could not insert probe at lnet_parse+0: -2 [ 6236.509985] Could not insert probe at lnet_parse+0: -2 [ 6246.751651] Could not insert probe at lnet_post_send_locked+0: -2 [ 6246.759232] Could not insert probe at lnet_post_send_locked+0: -2 [ 6257.011013] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6257.019935] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6267.249430] Could not insert probe at lnet_post_send_locked+0: -2 [ 6267.257109] Could not insert probe at lnet_post_send_locked+0: -2 [ 6277.502232] Could not insert probe at lnet_parse+0: -2 [ 6277.510936] Could not insert probe at lnet_parse+0: -2 [ 6287.751177] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6287.758816] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6298.005442] Could not insert probe at lnet_parse+0: -2 [ 6298.013214] Could not insert probe at lnet_parse+0: -2 [ 6308.249613] Could not insert probe at lnet_parse+0: -2 [ 6308.258764] Could not insert probe at lnet_parse+0: -2 [ 6318.503612] Could not insert probe at lnet_parse+0: -2 [ 6318.511041] Could not insert probe at lnet_parse+0: -2 [ 6328.750111] Could not insert probe at lnet_parse+0: -2 [ 6328.759409] Could not insert probe at lnet_parse+0: -2 [ 6339.003027] Could not insert probe at lnet_parse+0: -2 [ 6339.010843] Could not insert probe at lnet_parse+0: -2 [ 6349.254339] Could not insert probe at lnet_parse+0: -2 [ 6349.263305] Could not insert probe at lnet_parse+0: -2 [ 6359.505101] Could not insert probe at lnet_parse+0: -2 [ 6359.512682] Could not insert probe at lnet_parse+0: -2 [ 6369.752436] Could not insert probe at lnet_post_send_locked+0: -2 [ 6369.759923] Could not insert probe at lnet_post_send_locked+0: -2 [ 6380.004393] Could not insert probe at lnet_send+0: -2 [ 6380.013377] Could not insert probe at lnet_send+0: -2 [ 6390.260173] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6390.267884] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6400.512406] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6400.520230] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6410.758987] Could not insert probe at lnet_send+0: -2 [ 6410.766568] Could not insert probe at lnet_send+0: -2 [ 6421.006882] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6421.016078] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6431.255702] Could not insert probe at lnet_send+0: -2 [ 6431.263481] Could not insert probe at lnet_send+0: -2 [ 6441.511124] Could not insert probe at lnet_parse+0: -2 [ 6441.518788] Could not insert probe at lnet_parse+0: -2 [ 6451.758573] Could not insert probe at lnet_post_send_locked+0: -2 [ 6451.766263] Could not insert probe at lnet_post_send_locked+0: -2 [ 6462.010074] Could not insert probe at lnet_parse+0: -2 [ 6462.019566] Could not insert probe at lnet_parse+0: -2 [ 6472.262952] Could not insert probe at lnet_send+0: -2 [ 6472.272086] Could not insert probe at lnet_send+0: -2 [ 6482.507455] Could not insert probe at lnet_post_send_locked+0: -2 [ 6482.515415] Could not insert probe at lnet_post_send_locked+0: -2 [ 6492.760069] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6492.767850] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6503.009459] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6503.017139] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6513.258993] Could not insert probe at lnet_post_send_locked+0: -2 [ 6513.268641] Could not insert probe at lnet_post_send_locked+0: -2 [ 6523.507254] Could not insert probe at lnet_post_send_locked+0: -2 [ 6523.517052] Could not insert probe at lnet_post_send_locked+0: -2 [ 6533.763140] Could not insert probe at lnet_parse+0: -2 [ 6533.771031] Could not insert probe at lnet_parse+0: -2 [ 6544.016782] Could not insert probe at lnet_send+0: -2 [ 6544.025747] Could not insert probe at lnet_send+0: -2 [ 6554.258869] Could not insert probe at lnet_parse+0: -2 [ 6554.266610] Could not insert probe at lnet_parse+0: -2 [ 6564.508654] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6564.516490] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6574.757840] Could not insert probe at lnet_parse+0: -2 [ 6574.766879] Could not insert probe at lnet_parse+0: -2 [ 6585.023863] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6585.033144] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6595.264829] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6595.272773] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6605.519662] Could not insert probe at lnet_parse+0: -2 [ 6605.527167] Could not insert probe at lnet_parse+0: -2 [ 6615.769669] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6615.780124] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6626.013593] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6626.021713] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6636.264062] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6636.273409] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6646.516808] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6646.524640] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6656.764948] Could not insert probe at lnet_send+0: -2 [ 6656.772988] Could not insert probe at lnet_send+0: -2 [ 6667.014243] Could not insert probe at lnet_parse+0: -2 [ 6667.023544] Could not insert probe at lnet_parse+0: -2 [ 6677.272728] Could not insert probe at lnet_parse+0: -2 [ 6677.280991] Could not insert probe at lnet_parse+0: -2 [ 6687.515617] Could not insert probe at lnet_send+0: -2 [ 6687.523218] Could not insert probe at lnet_send+0: -2 [ 6697.770090] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6697.777989] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6708.016421] Could not insert probe at lnet_parse+0: -2 [ 6708.024179] Could not insert probe at lnet_parse+0: -2 [ 6718.266276] Could not insert probe at lnet_parse+0: -2 [ 6718.275507] Could not insert probe at lnet_parse+0: -2 [ 6728.515175] Could not insert probe at lnet_send+0: -2 [ 6728.522767] Could not insert probe at lnet_send+0: -2 [ 6738.767873] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6738.776142] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6749.019957] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6749.027683] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6759.268479] Could not insert probe at lnet_parse+0: -2 [ 6759.276090] Could not insert probe at lnet_parse+0: -2 [ 6769.518651] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6769.526545] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6779.781069] Could not insert probe at lnet_parse+0: -2 [ 6779.790932] Could not insert probe at lnet_parse+0: -2 [ 6790.022837] Could not insert probe at lnet_parse+0: -2 [ 6790.031340] Could not insert probe at lnet_parse+0: -2 [ 6800.270512] Could not insert probe at lnet_parse+0: -2 [ 6800.278525] Could not insert probe at lnet_parse+0: -2 [ 6810.523879] Could not insert probe at lnet_post_send_locked+0: -2 [ 6810.531838] Could not insert probe at lnet_post_send_locked+0: -2 [ 6820.774950] Could not insert probe at lnet_parse+0: -2 [ 6820.784558] Could not insert probe at lnet_parse+0: -2 [ 6831.022687] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6831.030889] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6841.271064] Could not insert probe at lnet_parse+0: -2 [ 6841.280737] Could not insert probe at lnet_parse+0: -2 [ 6851.530302] Could not insert probe at lnet_parse+0: -2 [ 6851.539771] Could not insert probe at lnet_parse+0: -2 [ 6861.774852] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6861.784086] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6872.031903] Could not insert probe at lnet_post_send_locked+0: -2 [ 6872.041500] Could not insert probe at lnet_post_send_locked+0: -2 [ 6882.283047] Could not insert probe at lnet_parse+0: -2 [ 6882.292974] Could not insert probe at lnet_parse+0: -2 [ 6892.523978] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6892.532380] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6902.775814] Could not insert probe at lnet_send+0: -2 [ 6902.783587] Could not insert probe at lnet_send+0: -2 [ 6913.034201] Could not insert probe at lnet_send+0: -2 [ 6913.043097] Could not insert probe at lnet_send+0: -2 [ 6923.276614] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6923.286567] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6933.532325] Could not insert probe at lnet_post_send_locked+0: -2 [ 6933.541451] Could not insert probe at lnet_post_send_locked+0: -2 [ 6943.783420] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6943.792911] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6954.034072] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6954.043426] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 6964.280972] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6964.289841] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 6974.538202] Could not insert probe at lnet_parse+0: -2 [ 6974.547539] Could not insert probe at lnet_parse+0: -2 [ 6984.784261] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6984.792743] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 6995.039155] Could not insert probe at lnet_parse+0: -2 [ 6995.048102] Could not insert probe at lnet_parse+0: -2 [ 7005.283475] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7005.292309] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7015.541147] Could not insert probe at lnet_parse+0: -2 [ 7015.551879] Could not insert probe at lnet_parse+0: -2 [ 7025.787755] Could not insert probe at lnet_parse+0: -2 [ 7025.797123] Could not insert probe at lnet_parse+0: -2 [ 7036.038648] Could not insert probe at lnet_parse+0: -2 [ 7036.047528] Could not insert probe at lnet_parse+0: -2 [ 7046.288423] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7046.298310] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7056.534366] Could not insert probe at lnet_parse+0: -2 [ 7056.543701] Could not insert probe at lnet_parse+0: -2 [ 7066.778105] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7066.787972] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7077.030657] Could not insert probe at lnet_send+0: -2 [ 7077.038460] Could not insert probe at lnet_send+0: -2 [ 7087.286980] Could not insert probe at lnet_parse+0: -2 [ 7087.295912] Could not insert probe at lnet_parse+0: -2 [ 7097.539279] Could not insert probe at lnet_parse+0: -2 [ 7097.549688] Could not insert probe at lnet_parse+0: -2 [ 7107.789203] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7107.797601] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7118.038832] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7118.048939] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7128.293961] Could not insert probe at lnet_post_send_locked+0: -2 [ 7128.303309] Could not insert probe at lnet_post_send_locked+0: -2 [ 7138.539390] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7138.548878] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7148.791899] Could not insert probe at lnet_parse+0: -2 [ 7148.801359] Could not insert probe at lnet_parse+0: -2 [ 7159.041680] Could not insert probe at lnet_parse+0: -2 [ 7159.051452] Could not insert probe at lnet_parse+0: -2 [ 7169.287502] Could not insert probe at lnet_parse+0: -2 [ 7169.297076] Could not insert probe at lnet_parse+0: -2 [ 7179.540848] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7179.550590] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7189.794799] Could not insert probe at lnet_post_send_locked+0: -2 [ 7189.804308] Could not insert probe at lnet_post_send_locked+0: -2 [ 7200.035287] Could not insert probe at lnet_parse+0: -2 [ 7200.043077] Could not insert probe at lnet_parse+0: -2 [ 7210.297535] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7210.308438] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7220.545073] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7220.555124] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7230.801078] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7230.810200] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7241.035440] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7241.044579] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7251.297403] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7251.307155] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7261.545050] Could not insert probe at lnet_parse+0: -2 [ 7261.554431] Could not insert probe at lnet_parse+0: -2 [ 7271.798943] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7271.809254] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7282.046241] Could not insert probe at lnet_post_send_locked+0: -2 [ 7282.055286] Could not insert probe at lnet_post_send_locked+0: -2 [ 7292.296974] Could not insert probe at lnet_post_send_locked+0: -2 [ 7292.306774] Could not insert probe at lnet_post_send_locked+0: -2 [ 7302.546602] Could not insert probe at lnet_parse+0: -2 [ 7302.555784] Could not insert probe at lnet_parse+0: -2 [ 7312.798397] Could not insert probe at lnet_send+0: -2 [ 7312.807239] Could not insert probe at lnet_send+0: -2 [ 7323.047610] Could not insert probe at lnet_parse+0: -2 [ 7323.056747] Could not insert probe at lnet_parse+0: -2 [ 7333.296613] Could not insert probe at lnet_parse+0: -2 [ 7333.306253] Could not insert probe at lnet_parse+0: -2 [ 7343.549475] Could not insert probe at lnet_parse+0: -2 [ 7343.558461] Could not insert probe at lnet_parse+0: -2 [ 7353.798998] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7353.809949] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7364.048487] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7364.059389] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7374.297906] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7374.308063] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7384.551416] Could not insert probe at lnet_parse+0: -2 [ 7384.561294] Could not insert probe at lnet_parse+0: -2 [ 7394.799687] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7394.809365] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7405.054190] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7405.063927] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7415.295346] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7415.304550] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7425.553570] Could not insert probe at lnet_parse+0: -2 [ 7425.562726] Could not insert probe at lnet_parse+0: -2 [ 7435.801358] Could not insert probe at lnet_parse+0: -2 [ 7435.812204] Could not insert probe at lnet_parse+0: -2 [ 7446.056043] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7446.065487] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7456.298713] Could not insert probe at lnet_parse+0: -2 [ 7456.308093] Could not insert probe at lnet_parse+0: -2 [ 7466.556830] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7466.566183] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7476.803461] Could not insert probe at lnet_parse+0: -2 [ 7476.812796] Could not insert probe at lnet_parse+0: -2 [ 7487.058105] Could not insert probe at lnet_parse+0: -2 [ 7487.067005] Could not insert probe at lnet_parse+0: -2 [ 7497.303888] Could not insert probe at lnet_parse+0: -2 [ 7497.314574] Could not insert probe at lnet_parse+0: -2 [ 7507.557698] Could not insert probe at lnet_post_send_locked+0: -2 [ 7507.566964] Could not insert probe at lnet_post_send_locked+0: -2 [ 7517.806224] Could not insert probe at lnet_parse+0: -2 [ 7517.815511] Could not insert probe at lnet_parse+0: -2 [ 7528.057025] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7528.066734] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7538.305589] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7538.315205] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7548.556777] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7548.566503] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7558.805954] Could not insert probe at lnet_parse+0: -2 [ 7558.814962] Could not insert probe at lnet_parse+0: -2 [ 7569.053107] Could not insert probe at lnet_send+0: -2 [ 7569.062592] Could not insert probe at lnet_send+0: -2 [ 7579.304539] Could not insert probe at lnet_parse+0: -2 [ 7579.313862] Could not insert probe at lnet_parse+0: -2 [ 7589.552133] Could not insert probe at lnet_parse+0: -2 [ 7589.560232] Could not insert probe at lnet_parse+0: -2 [ 7599.811320] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7599.820794] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7610.050472] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7610.058577] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7620.305714] Could not insert probe at lnet_parse+0: -2 [ 7620.313332] Could not insert probe at lnet_parse+0: -2 [ 7630.560064] Could not insert probe at lnet_parse+0: -2 [ 7630.569882] Could not insert probe at lnet_parse+0: -2 [ 7640.815458] Could not insert probe at lnet_send+0: -2 [ 7640.826156] Could not insert probe at lnet_send+0: -2 [ 7651.055871] Could not insert probe at lnet_post_send_locked+0: -2 [ 7651.063541] Could not insert probe at lnet_post_send_locked+0: -2 [ 7661.315103] Could not insert probe at lnet_post_send_locked+0: -2 [ 7661.323097] Could not insert probe at lnet_post_send_locked+0: -2 [ 7671.561172] Could not insert probe at lnet_post_send_locked+0: -2 [ 7671.570459] Could not insert probe at lnet_post_send_locked+0: -2 [ 7681.817985] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7681.827522] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7692.059860] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7692.069007] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7702.310854] Could not insert probe at lnet_send+0: -2 [ 7702.318618] Could not insert probe at lnet_send+0: -2 [ 7712.564344] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7712.573932] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7722.804308] Could not insert probe at lnet_parse+0: -2 [ 7722.812247] Could not insert probe at lnet_parse+0: -2 [ 7733.056281] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7733.064639] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7743.314921] Could not insert probe at lnet_send+0: -2 [ 7743.323959] Could not insert probe at lnet_send+0: -2 [ 7753.566033] Could not insert probe at lnet_post_send_locked+0: -2 [ 7753.575818] Could not insert probe at lnet_post_send_locked+0: -2 [ 7763.807228] Could not insert probe at lnet_parse+0: -2 [ 7763.814988] Could not insert probe at lnet_parse+0: -2 [ 7774.057181] Could not insert probe at lnet_parse+0: -2 [ 7774.066343] Could not insert probe at lnet_parse+0: -2 [ 7784.315393] Could not insert probe at lnet_send+0: -2 [ 7784.324771] Could not insert probe at lnet_send+0: -2 [ 7794.567492] Could not insert probe at lnet_parse+0: -2 [ 7794.577324] Could not insert probe at lnet_parse+0: -2 [ 7804.815993] Could not insert probe at lnet_send+0: -2 [ 7804.825446] Could not insert probe at lnet_send+0: -2 [ 7815.068356] Could not insert probe at lnet_post_send_locked+0: -2 [ 7815.077702] Could not insert probe at lnet_post_send_locked+0: -2 [ 7825.316966] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7825.326396] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7835.570454] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7835.579922] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7845.817580] Could not insert probe at lnet_post_send_locked+0: -2 [ 7845.826883] Could not insert probe at lnet_post_send_locked+0: -2 [ 7856.070392] Could not insert probe at lnet_parse+0: -2 [ 7856.078420] Could not insert probe at lnet_parse+0: -2 [ 7866.318247] Could not insert probe at lnet_parse+0: -2 [ 7866.327625] Could not insert probe at lnet_parse+0: -2 [ 7876.573552] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7876.583094] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7886.818707] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7886.828665] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 7897.074302] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7897.084014] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7907.320731] Could not insert probe at lnet_parse+0: -2 [ 7907.328443] Could not insert probe at lnet_parse+0: -2 [ 7917.572137] Could not insert probe at lnet_parse+0: -2 [ 7917.582210] Could not insert probe at lnet_parse+0: -2 [ 7927.820712] Could not insert probe at lnet_parse+0: -2 [ 7927.829976] Could not insert probe at lnet_parse+0: -2 [ 7938.064859] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7938.074536] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 7948.324451] Could not insert probe at lnet_parse+0: -2 [ 7948.334142] Could not insert probe at lnet_parse+0: -2 [ 7958.572740] Could not insert probe at lnet_send+0: -2 [ 7958.580535] Could not insert probe at lnet_send+0: -2 [ 7968.825350] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7968.834933] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 7979.071865] Could not insert probe at lnet_parse+0: -2 [ 7979.086820] Could not insert probe at lnet_parse+0: -2 [ 7989.324978] Could not insert probe at lnet_parse+0: -2 [ 7989.334722] Could not insert probe at lnet_parse+0: -2 [ 7999.572976] Could not insert probe at lnet_parse+0: -2 [ 7999.582147] Could not insert probe at lnet_parse+0: -2 [ 8009.827975] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8009.837433] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8020.068209] Could not insert probe at lnet_parse+0: -2 [ 8020.077736] Could not insert probe at lnet_parse+0: -2 [ 8030.319649] Could not insert probe at lnet_parse+0: -2 [ 8030.327209] Could not insert probe at lnet_parse+0: -2 [ 8040.575477] Could not insert probe at lnet_parse+0: -2 [ 8040.584769] Could not insert probe at lnet_parse+0: -2 [ 8050.829263] Could not insert probe at lnet_parse+0: -2 [ 8050.839135] Could not insert probe at lnet_parse+0: -2 [ 8061.076088] Could not insert probe at lnet_parse+0: -2 [ 8061.085325] Could not insert probe at lnet_parse+0: -2 [ 8071.330247] Could not insert probe at lnet_post_send_locked+0: -2 [ 8071.340014] Could not insert probe at lnet_post_send_locked+0: -2 [ 8081.577341] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8081.585340] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8091.820436] Could not insert probe at lnet_post_send_locked+0: -2 [ 8091.828423] Could not insert probe at lnet_post_send_locked+0: -2 [ 8102.080952] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8102.091860] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8112.331068] Could not insert probe at lnet_send+0: -2 [ 8112.340572] Could not insert probe at lnet_send+0: -2 [ 8122.575957] Could not insert probe at lnet_parse+0: -2 [ 8122.586559] Could not insert probe at lnet_parse+0: -2 [ 8132.833809] Could not insert probe at lnet_parse+0: -2 [ 8132.843203] Could not insert probe at lnet_parse+0: -2 [ 8143.077435] Could not insert probe at lnet_parse+0: -2 [ 8143.085747] Could not insert probe at lnet_parse+0: -2 [ 8153.324635] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8153.332740] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8163.580580] Could not insert probe at lnet_parse+0: -2 [ 8163.590215] Could not insert probe at lnet_parse+0: -2 [ 8173.830728] Could not insert probe at lnet_parse+0: -2 [ 8173.842038] Could not insert probe at lnet_parse+0: -2 [ 8184.079915] Could not insert probe at lnet_send+0: -2 [ 8184.088095] Could not insert probe at lnet_send+0: -2 [ 8194.323566] Could not insert probe at lnet_send+0: -2 [ 8194.334003] Could not insert probe at lnet_send+0: -2 [ 8204.584404] Could not insert probe at lnet_parse+0: -2 [ 8204.594006] Could not insert probe at lnet_parse+0: -2 [ 8214.832067] Could not insert probe at lnet_parse+0: -2 [ 8214.841154] Could not insert probe at lnet_parse+0: -2 [ 8225.082656] Could not insert probe at lnet_parse+0: -2 [ 8225.091817] Could not insert probe at lnet_parse+0: -2 [ 8235.332249] Could not insert probe at lnet_post_send_locked+0: -2 [ 8235.342366] Could not insert probe at lnet_post_send_locked+0: -2 [ 8245.587593] Could not insert probe at lnet_parse+0: -2 [ 8245.596638] Could not insert probe at lnet_parse+0: -2 [ 8255.833087] Could not insert probe at lnet_parse+0: -2 [ 8255.843839] Could not insert probe at lnet_parse+0: -2 [ 8266.088359] Could not insert probe at lnet_parse+0: -2 [ 8266.097277] Could not insert probe at lnet_parse+0: -2 [ 8276.334114] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8276.343821] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8286.587686] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8286.596847] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8296.834565] Could not insert probe at lnet_parse+0: -2 [ 8296.842418] Could not insert probe at lnet_parse+0: -2 [ 8307.090046] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8307.099794] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8317.336015] Could not insert probe at lnet_parse+0: -2 [ 8317.345432] Could not insert probe at lnet_parse+0: -2 [ 8327.590686] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8327.599587] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8337.831808] Could not insert probe at lnet_parse+0: -2 [ 8337.839933] Could not insert probe at lnet_parse+0: -2 [ 8348.085447] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8348.093483] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8358.336355] Could not insert probe at lnet_parse+0: -2 [ 8358.345959] Could not insert probe at lnet_parse+0: -2 [ 8368.578035] Could not insert probe at lnet_post_send_locked+0: -2 [ 8368.585714] Could not insert probe at lnet_post_send_locked+0: -2 [ 8378.840527] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8378.851788] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8389.089878] Could not insert probe at lnet_parse+0: -2 [ 8389.098864] Could not insert probe at lnet_parse+0: -2 [ 8399.341830] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8399.351840] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8409.590193] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8409.599747] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8419.844107] Could not insert probe at lnet_parse+0: -2 [ 8419.853082] Could not insert probe at lnet_parse+0: -2 [ 8430.089617] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8430.100510] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8440.343938] Could not insert probe at lnet_send+0: -2 [ 8440.352153] Could not insert probe at lnet_send+0: -2 [ 8450.581323] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8450.589130] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8460.846672] Could not insert probe at lnet_parse+0: -2 [ 8460.856077] Could not insert probe at lnet_parse+0: -2 [ 8471.091342] Could not insert probe at lnet_send+0: -2 [ 8471.100061] Could not insert probe at lnet_send+0: -2 [ 8481.347062] Could not insert probe at lnet_post_send_locked+0: -2 [ 8481.356741] Could not insert probe at lnet_post_send_locked+0: -2 [ 8491.591590] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8491.601154] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8501.850308] Could not insert probe at lnet_parse+0: -2 [ 8501.858110] Could not insert probe at lnet_parse+0: -2 [ 8512.093270] Could not insert probe at lnet_parse+0: -2 [ 8512.102893] Could not insert probe at lnet_parse+0: -2 [ 8522.348266] Could not insert probe at lnet_parse+0: -2 [ 8522.359410] Could not insert probe at lnet_parse+0: -2 [ 8532.592828] Could not insert probe at lnet_parse+0: -2 [ 8532.603588] Could not insert probe at lnet_parse+0: -2 [ 8542.847240] Could not insert probe at lnet_send+0: -2 [ 8542.856864] Could not insert probe at lnet_send+0: -2 [ 8553.092743] Could not insert probe at lnet_send+0: -2 [ 8553.102436] Could not insert probe at lnet_send+0: -2 [ 8563.347354] Could not insert probe at lnet_parse+0: -2 [ 8563.356932] Could not insert probe at lnet_parse+0: -2 [ 8573.597090] Could not insert probe at lnet_post_send_locked+0: -2 [ 8573.605406] Could not insert probe at lnet_post_send_locked+0: -2 [ 8583.846640] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8583.856437] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8594.096144] Could not insert probe at lnet_parse+0: -2 [ 8594.104022] Could not insert probe at lnet_parse+0: -2 [ 8604.345327] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8604.356214] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8614.594471] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8614.602677] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8624.839766] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8624.849290] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8635.098047] Could not insert probe at lnet_parse+0: -2 [ 8635.107717] Could not insert probe at lnet_parse+0: -2 [ 8645.338755] Could not insert probe at lnet_post_send_locked+0: -2 [ 8645.348094] Could not insert probe at lnet_post_send_locked+0: -2 [ 8655.602459] Could not insert probe at lnet_parse+0: -2 [ 8655.613296] Could not insert probe at lnet_parse+0: -2 [ 8665.849211] Could not insert probe at lnet_post_send_locked+0: -2 [ 8665.858680] Could not insert probe at lnet_post_send_locked+0: -2 [ 8676.104572] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8676.114027] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8686.349707] Could not insert probe at lnet_parse+0: -2 [ 8686.360403] Could not insert probe at lnet_parse+0: -2 [ 8696.605842] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8696.616817] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8706.851378] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8706.862290] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8717.105696] Could not insert probe at lnet_send+0: -2 [ 8717.114712] Could not insert probe at lnet_send+0: -2 [ 8727.352151] Could not insert probe at lnet_parse+0: -2 [ 8727.361132] Could not insert probe at lnet_parse+0: -2 [ 8737.607159] Could not insert probe at lnet_send+0: -2 [ 8737.616476] Could not insert probe at lnet_send+0: -2 [ 8747.851543] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8747.860807] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8758.107131] Could not insert probe at lnet_post_send_locked+0: -2 [ 8758.116108] Could not insert probe at lnet_post_send_locked+0: -2 [ 8768.351959] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8768.361735] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8778.603681] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8778.611841] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8788.853651] Could not insert probe at lnet_parse+0: -2 [ 8788.863443] Could not insert probe at lnet_parse+0: -2 [ 8799.105881] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8799.116251] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8809.353373] Could not insert probe at lnet_parse+0: -2 [ 8809.361223] Could not insert probe at lnet_parse+0: -2 [ 8819.602770] Could not insert probe at lnet_post_send_locked+0: -2 [ 8819.612799] Could not insert probe at lnet_post_send_locked+0: -2 [ 8829.856322] Could not insert probe at lnet_parse+0: -2 [ 8829.865708] Could not insert probe at lnet_parse+0: -2 [ 8840.106811] Could not insert probe at lnet_parse+0: -2 [ 8840.116294] Could not insert probe at lnet_parse+0: -2 [ 8850.353460] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8850.362884] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8860.605650] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8860.615218] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 8870.861962] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8870.871790] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8881.099906] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8881.110482] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8891.363106] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8891.372443] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8901.607629] Could not insert probe at lnet_parse+0: -2 [ 8901.618108] Could not insert probe at lnet_parse+0: -2 [ 8911.865884] Could not insert probe at lnet_send+0: -2 [ 8911.875272] Could not insert probe at lnet_send+0: -2 [ 8922.109161] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8922.118853] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8932.362706] Could not insert probe at lnet_parse+0: -2 [ 8932.372270] Could not insert probe at lnet_parse+0: -2 [ 8942.608263] Could not insert probe at lnet_send+0: -2 [ 8942.616312] Could not insert probe at lnet_send+0: -2 [ 8952.862285] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8952.871624] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 8963.110908] Could not insert probe at lnet_post_send_locked+0: -2 [ 8963.120328] Could not insert probe at lnet_post_send_locked+0: -2 [ 8973.352064] Could not insert probe at lnet_parse+0: -2 [ 8973.360965] Could not insert probe at lnet_parse+0: -2 [ 8983.603692] Could not insert probe at lnet_post_send_locked+0: -2 [ 8983.612813] Could not insert probe at lnet_post_send_locked+0: -2 [ 8993.864362] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 8993.874127] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9004.110052] Could not insert probe at lnet_post_send_locked+0: -2 [ 9004.119861] Could not insert probe at lnet_post_send_locked+0: -2 [ 9014.358149] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9014.368001] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9024.612441] Could not insert probe at lnet_parse+0: -2 [ 9024.621413] Could not insert probe at lnet_parse+0: -2 [ 9034.862443] Could not insert probe at lnet_post_send_locked+0: -2 [ 9034.871997] Could not insert probe at lnet_post_send_locked+0: -2 [ 9045.112164] Could not insert probe at lnet_post_send_locked+0: -2 [ 9045.120347] Could not insert probe at lnet_post_send_locked+0: -2 [ 9055.357466] Could not insert probe at lnet_post_send_locked+0: -2 [ 9055.366702] Could not insert probe at lnet_post_send_locked+0: -2 [ 9065.617535] Could not insert probe at lnet_parse+0: -2 [ 9065.625026] Could not insert probe at lnet_parse+0: -2 [ 9075.863252] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9075.871516] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9086.123349] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9086.131735] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9096.365193] Could not insert probe at lnet_parse+0: -2 [ 9096.374450] Could not insert probe at lnet_parse+0: -2 [ 9106.619777] Could not insert probe at lnet_post_send_locked+0: -2 [ 9106.627673] Could not insert probe at lnet_post_send_locked+0: -2 [ 9116.860554] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9116.870139] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9127.137355] Could not insert probe at lnet_parse+0: -2 [ 9127.148112] Could not insert probe at lnet_parse+0: -2 [ 9137.389600] Could not insert probe at lnet_parse+0: -2 [ 9137.398559] Could not insert probe at lnet_parse+0: -2 [ 9147.625616] Could not insert probe at lnet_parse+0: -2 [ 9147.635073] Could not insert probe at lnet_parse+0: -2 [ 9157.882096] Could not insert probe at lnet_parse+0: -2 [ 9157.892075] Could not insert probe at lnet_parse+0: -2 [ 9168.135254] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9168.145837] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9178.383712] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9178.393343] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9188.634019] Could not insert probe at lnet_parse+0: -2 [ 9188.644679] Could not insert probe at lnet_parse+0: -2 [ 9198.882926] Could not insert probe at lnet_parse+0: -2 [ 9198.893154] Could not insert probe at lnet_parse+0: -2 [ 9209.136065] Could not insert probe at lnet_send+0: -2 [ 9209.145649] Could not insert probe at lnet_send+0: -2 [ 9219.384034] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9219.394694] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9229.637475] Could not insert probe at lnet_parse+0: -2 [ 9229.648471] Could not insert probe at lnet_parse+0: -2 [ 9239.887256] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9239.896566] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9250.140964] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9250.152815] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9260.394265] Could not insert probe at lnet_parse+0: -2 [ 9260.404509] Could not insert probe at lnet_parse+0: -2 [ 9270.617693] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9270.625847] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9280.869745] Could not insert probe at lnet_parse+0: -2 [ 9280.877386] Could not insert probe at lnet_parse+0: -2 [ 9291.117979] Could not insert probe at lnet_parse+0: -2 [ 9291.127773] Could not insert probe at lnet_parse+0: -2 [ 9301.381368] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9301.391295] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9311.627913] Could not insert probe at lnet_parse+0: -2 [ 9311.635585] Could not insert probe at lnet_parse+0: -2 [ 9321.863620] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9321.871959] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9332.119948] Could not insert probe at lnet_send+0: -2 [ 9332.128307] Could not insert probe at lnet_send+0: -2 [ 9342.373191] Could not insert probe at lnet_parse+0: -2 [ 9342.382696] Could not insert probe at lnet_parse+0: -2 [ 9352.622011] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9352.630005] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9362.872818] Could not insert probe at lnet_parse+0: -2 [ 9362.880709] Could not insert probe at lnet_parse+0: -2 [ 9373.117103] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9373.127482] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9383.370093] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9383.378242] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9393.623906] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9393.631560] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9403.873833] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9403.881875] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9414.126699] Could not insert probe at lnet_parse+0: -2 [ 9414.134414] Could not insert probe at lnet_parse+0: -2 [ 9424.378274] Could not insert probe at lnet_parse+0: -2 [ 9424.388011] Could not insert probe at lnet_parse+0: -2 [ 9434.627253] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9434.635349] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9444.875095] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9444.883624] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9455.119721] Could not insert probe at lnet_post_send_locked+0: -2 [ 9455.129036] Could not insert probe at lnet_post_send_locked+0: -2 [ 9465.372760] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9465.380439] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9475.628627] Could not insert probe at lnet_parse+0: -2 [ 9475.636642] Could not insert probe at lnet_parse+0: -2 [ 9485.874446] Could not insert probe at lnet_send+0: -2 [ 9485.882342] Could not insert probe at lnet_send+0: -2 [ 9496.125796] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9496.133646] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9506.380468] Could not insert probe at lnet_parse+0: -2 [ 9506.390022] Could not insert probe at lnet_parse+0: -2 [ 9516.636387] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9516.647327] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [ 9526.876705] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9526.884834] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9537.126260] Could not insert probe at lnet_post_send_locked+0: -2 [ 9537.134058] Could not insert probe at lnet_post_send_locked+0: -2 [ 9547.380966] Could not insert probe at lnet_parse+0: -2 [ 9547.390558] Could not insert probe at lnet_parse+0: -2 [ 9557.631096] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9557.639008] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9567.885357] Could not insert probe at lnet_post_send_locked+0: -2 [ 9567.893369] Could not insert probe at lnet_post_send_locked+0: -2 [ 9578.124441] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9578.132169] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9588.377973] Could not insert probe at lnet_post_send_locked+0: -2 [ 9588.386099] Could not insert probe at lnet_post_send_locked+0: -2 [ 9598.630035] Could not insert probe at lnet_parse+0: -2 [ 9598.637432] Could not insert probe at lnet_parse+0: -2 [ 9608.876565] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9608.884896] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9619.124970] Could not insert probe at lnet_parse+0: -2 [ 9619.134307] Could not insert probe at lnet_parse+0: -2 [ 9629.384182] Could not insert probe at lnet_parse+0: -2 [ 9629.391744] Could not insert probe at lnet_parse+0: -2 [ 9639.635034] Could not insert probe at lnet_parse+0: -2 [ 9639.643138] Could not insert probe at lnet_parse+0: -2 [ 9649.882624] Could not insert probe at lnet_parse+0: -2 [ 9649.891486] Could not insert probe at lnet_parse+0: -2 [ 9660.128353] Could not insert probe at lnet_parse+0: -2 [ 9660.135804] Could not insert probe at lnet_parse+0: -2 [ 9670.378107] Could not insert probe at lnet_send+0: -2 [ 9670.387388] Could not insert probe at lnet_send+0: -2 [ 9680.633476] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9680.641508] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9690.884958] Could not insert probe at lnet_send+0: -2 [ 9690.893092] Could not insert probe at lnet_send+0: -2 [ 9701.133046] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9701.142571] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9711.387795] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9711.395851] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9721.634680] Could not insert probe at lnet_parse+0: -2 [ 9721.642261] Could not insert probe at lnet_parse+0: -2 [ 9731.886246] Could not insert probe at lnet_send+0: -2 [ 9731.895764] Could not insert probe at lnet_send+0: -2 [ 9742.131318] Could not insert probe at lnet_parse+0: -2 [ 9742.139044] Could not insert probe at lnet_parse+0: -2 [ 9752.387349] Could not insert probe at lnet_post_send_locked+0: -2 [ 9752.396482] Could not insert probe at lnet_post_send_locked+0: -2 [ 9762.633250] Could not insert probe at lnet_post_send_locked+0: -2 [ 9762.640869] Could not insert probe at lnet_post_send_locked+0: -2 [ 9772.885490] Could not insert probe at lnet_parse+0: -2 [ 9772.893267] Could not insert probe at lnet_parse+0: -2 [ 9783.133682] Could not insert probe at lnet_parse+0: -2 [ 9783.142554] Could not insert probe at lnet_parse+0: -2 [ 9793.384622] Could not insert probe at lnet_parse+0: -2 [ 9793.391765] Could not insert probe at lnet_parse+0: -2 [ 9803.637242] Could not insert probe at lnet_parse+0: -2 [ 9803.645456] Could not insert probe at lnet_parse+0: -2 [ 9813.885996] Could not insert probe at lnet_parse+0: -2 [ 9813.893849] Could not insert probe at lnet_parse+0: -2 [ 9824.144287] Could not insert probe at lnet_parse+0: -2 [ 9824.152353] Could not insert probe at lnet_parse+0: -2 [ 9834.388054] Could not insert probe at lnet_post_send_locked+0: -2 [ 9834.397563] Could not insert probe at lnet_post_send_locked+0: -2 [ 9844.642263] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9844.650056] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9854.886395] Could not insert probe at lnet_parse+0: -2 [ 9854.894297] Could not insert probe at lnet_parse+0: -2 [ 9865.149660] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9865.158840] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [ 9875.393606] Could not insert probe at lnet_post_send_locked+0: -2 [ 9875.401240] Could not insert probe at lnet_post_send_locked+0: -2 [ 9885.641598] Could not insert probe at lnet_send+0: -2 [ 9885.649614] Could not insert probe at lnet_send+0: -2 [ 9895.896981] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9895.905042] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9906.153627] Could not insert probe at lnet_send+0: -2 [ 9906.164360] Could not insert probe at lnet_send+0: -2 [ 9916.396291] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9916.405740] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9926.644393] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9926.652117] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9936.894428] Could not insert probe at lnet_parse+0: -2 [ 9936.902600] Could not insert probe at lnet_parse+0: -2 [ 9947.139767] Could not insert probe at lnet_parse+0: -2 [ 9947.147580] Could not insert probe at lnet_parse+0: -2 [ 9957.392593] Could not insert probe at lnet_post_send_locked+0: -2 [ 9957.402364] Could not insert probe at lnet_post_send_locked+0: -2 [ 9967.650856] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9967.660775] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9977.892916] Could not insert probe at lnet_parse+0: -2 [ 9977.901177] Could not insert probe at lnet_parse+0: -2 [ 9988.154973] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9988.163272] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [ 9998.404497] Could not insert probe at lnet_parse+0: -2 [ 9998.413016] Could not insert probe at lnet_parse+0: -2 [10008.646345] Could not insert probe at lnet_send+0: -2 [10008.654429] Could not insert probe at lnet_send+0: -2 [10018.895753] Could not insert probe at lnet_parse+0: -2 [10018.903947] Could not insert probe at lnet_parse+0: -2 [10029.143906] Could not insert probe at lnet_parse+0: -2 [10029.151770] Could not insert probe at lnet_parse+0: -2 [10039.397999] Could not insert probe at lnet_parse+0: -2 [10039.407490] Could not insert probe at lnet_parse+0: -2 [10049.642674] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10049.650801] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10059.901615] Could not insert probe at lnet_parse+0: -2 [10059.909353] Could not insert probe at lnet_parse+0: -2 [10070.150857] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10070.158729] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10080.399085] Could not insert probe at lnet_parse+0: -2 [10080.407071] Could not insert probe at lnet_parse+0: -2 [10090.654440] Could not insert probe at lnet_post_send_locked+0: -2 [10090.663524] Could not insert probe at lnet_post_send_locked+0: -2 [10100.904591] Could not insert probe at lnet_send+0: -2 [10100.912849] Could not insert probe at lnet_send+0: -2 [10111.149466] Could not insert probe at lnet_parse+0: -2 [10111.157417] Could not insert probe at lnet_parse+0: -2 [10121.409281] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10121.418646] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10131.652427] Could not insert probe at lnet_post_send_locked+0: -2 [10131.660105] Could not insert probe at lnet_post_send_locked+0: -2 [10141.901186] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10141.909511] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10152.157490] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10152.167008] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10162.412823] Could not insert probe at lnet_parse+0: -2 [10162.421455] Could not insert probe at lnet_parse+0: -2 [10172.657620] Could not insert probe at lnet_send+0: -2 [10172.666661] Could not insert probe at lnet_send+0: -2 [10182.905334] Could not insert probe at lnet_post_send_locked+0: -2 [10182.913281] Could not insert probe at lnet_post_send_locked+0: -2 [10193.148455] Could not insert probe at lnet_post_send_locked+0: -2 [10193.157367] Could not insert probe at lnet_post_send_locked+0: -2 [10203.411698] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10203.421821] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10213.659863] Could not insert probe at lnet_parse+0: -2 [10213.669221] Could not insert probe at lnet_parse+0: -2 [10223.909339] Could not insert probe at lnet_parse+0: -2 [10223.918612] Could not insert probe at lnet_parse+0: -2 [10234.159614] Could not insert probe at lnet_parse+0: -2 [10234.168107] Could not insert probe at lnet_parse+0: -2 [10244.402939] Could not insert probe at lnet_parse+0: -2 [10244.411702] Could not insert probe at lnet_parse+0: -2 [10254.660563] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10254.669014] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10264.909042] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10264.918224] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10275.152996] Could not insert probe at lnet_parse+0: -2 [10275.162529] Could not insert probe at lnet_parse+0: -2 [10285.409820] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10285.417907] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10295.658402] Could not insert probe at lnet_parse+0: -2 [10295.669060] Could not insert probe at lnet_parse+0: -2 [10305.911013] Could not insert probe at lnet_send+0: -2 [10305.918678] Could not insert probe at lnet_send+0: -2 [10316.166952] Could not insert probe at lnet_post_send_locked+0: -2 [10316.175919] Could not insert probe at lnet_post_send_locked+0: -2 [10326.412548] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10326.420464] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10336.668382] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10336.677937] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10346.910722] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10346.920442] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10357.166604] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10357.175348] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10367.415120] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10367.423070] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10377.668578] Could not insert probe at lnet_parse+0: -2 [10377.679433] Could not insert probe at lnet_parse+0: -2 [10387.915095] Could not insert probe at lnet_post_send_locked+0: -2 [10387.924911] Could not insert probe at lnet_post_send_locked+0: -2 [10398.157264] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10398.167639] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10408.415969] Could not insert probe at lnet_parse+0: -2 [10408.425871] Could not insert probe at lnet_parse+0: -2 [10418.661678] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10418.669717] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10428.911205] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10428.919502] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10439.170642] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10439.180632] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10449.419501] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10449.429486] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10459.668899] Could not insert probe at lnet_parse+0: -2 [10459.679011] Could not insert probe at lnet_parse+0: -2 [10469.914714] Could not insert probe at lnet_send+0: -2 [10469.922184] Could not insert probe at lnet_send+0: -2 [10480.166335] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10480.174517] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10490.417805] Could not insert probe at lnet_parse+0: -2 [10490.426777] Could not insert probe at lnet_parse+0: -2 [10500.668310] Could not insert probe at lnet_parse+0: -2 [10500.678331] Could not insert probe at lnet_parse+0: -2 [10510.923602] Could not insert probe at lnet_parse+0: -2 [10510.933030] Could not insert probe at lnet_parse+0: -2 [10521.161855] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10521.170999] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10531.425382] Could not insert probe at lnet_post_send_locked+0: -2 [10531.436249] Could not insert probe at lnet_post_send_locked+0: -2 [10541.671401] Could not insert probe at lnet_post_send_locked+0: -2 [10541.680788] Could not insert probe at lnet_post_send_locked+0: -2 [10551.914963] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10551.923654] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10562.168137] Could not insert probe at lnet_parse+0: -2 [10562.177764] Could not insert probe at lnet_parse+0: -2 [10572.425047] Could not insert probe at lnet_parse+0: -2 [10572.435791] Could not insert probe at lnet_parse+0: -2 [10582.672051] Could not insert probe at lnet_parse+0: -2 [10582.681631] Could not insert probe at lnet_parse+0: -2 [10592.929586] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10592.938752] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10603.163683] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10603.173152] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10613.427030] Could not insert probe at lnet_parse+0: -2 [10613.436037] Could not insert probe at lnet_parse+0: -2 [10623.673457] Could not insert probe at lnet_parse+0: -2 [10623.683081] Could not insert probe at lnet_parse+0: -2 [10633.913914] Could not insert probe at lnet_send+0: -2 [10633.922828] Could not insert probe at lnet_send+0: -2 [10644.175039] Could not insert probe at lnet_parse+0: -2 [10644.184815] Could not insert probe at lnet_parse+0: -2 [10654.426949] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10654.438407] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10664.675130] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10664.684437] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10674.927322] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10674.937524] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10685.176742] Could not insert probe at lnet_send+0: -2 [10685.186144] Could not insert probe at lnet_send+0: -2 [10695.426980] Could not insert probe at lnet_parse+0: -2 [10695.436660] Could not insert probe at lnet_parse+0: -2 [10705.667376] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10705.677321] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10715.927306] Could not insert probe at lnet_parse+0: -2 [10715.936521] Could not insert probe at lnet_parse+0: -2 [10726.182810] Could not insert probe at lnet_parse+0: -2 [10726.191002] Could not insert probe at lnet_parse+0: -2 [10736.429856] Could not insert probe at lnet_post_send_locked+0: -2 [10736.439287] Could not insert probe at lnet_post_send_locked+0: -2 [10746.685605] Could not insert probe at lnet_parse+0: -2 [10746.694555] Could not insert probe at lnet_parse+0: -2 [10756.928856] Could not insert probe at lnet_send+0: -2 [10756.938156] Could not insert probe at lnet_send+0: -2 [10767.184293] Could not insert probe at lnet_parse+0: -2 [10767.193218] Could not insert probe at lnet_parse+0: -2 [10777.429183] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10777.439019] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10787.683882] Could not insert probe at lnet_parse+0: -2 [10787.693281] Could not insert probe at lnet_parse+0: -2 [10797.931383] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10797.940412] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10808.183545] Could not insert probe at lnet_parse+0: -2 [10808.192355] Could not insert probe at lnet_parse+0: -2 [10818.427767] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10818.436383] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [10828.685488] Could not insert probe at lnet_parse+0: -2 [10828.694701] Could not insert probe at lnet_parse+0: -2 [10838.932715] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10838.942157] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10849.185552] Could not insert probe at lnet_post_send_locked+0: -2 [10849.195593] Could not insert probe at lnet_post_send_locked+0: -2 [10859.433750] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10859.442951] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10869.685133] Could not insert probe at lnet_parse+0: -2 [10869.694168] Could not insert probe at lnet_parse+0: -2 [10879.935114] Could not insert probe at lnet_parse+0: -2 [10879.944691] Could not insert probe at lnet_parse+0: -2 [10890.185115] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10890.196564] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [10900.435489] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10900.444666] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10910.685655] Could not insert probe at lnet_parse+0: -2 [10910.694085] Could not insert probe at lnet_parse+0: -2 [10920.938617] Could not insert probe at lnet_send+0: -2 [10920.948497] Could not insert probe at lnet_send+0: -2 [10931.179559] Could not insert probe at lnet_send+0: -2 [10931.188317] Could not insert probe at lnet_send+0: -2 [10941.432711] Could not insert probe at lnet_send+0: -2 [10941.442232] Could not insert probe at lnet_send+0: -2 [10951.687568] Could not insert probe at lnet_parse+0: -2 [10951.696701] Could not insert probe at lnet_parse+0: -2 [10961.942770] Could not insert probe at lnet_parse+0: -2 [10961.952264] Could not insert probe at lnet_parse+0: -2 [10972.178388] Could not insert probe at lnet_parse+0: -2 [10972.186387] Could not insert probe at lnet_parse+0: -2 [10982.444774] Could not insert probe at lnet_parse+0: -2 [10982.453771] Could not insert probe at lnet_parse+0: -2 [10992.684976] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [10992.694121] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11002.942366] Could not insert probe at lnet_post_send_locked+0: -2 [11002.951515] Could not insert probe at lnet_post_send_locked+0: -2 [11013.189272] Could not insert probe at lnet_send+0: -2 [11013.198857] Could not insert probe at lnet_send+0: -2 [11023.440780] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11023.450608] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11033.690159] Could not insert probe at lnet_parse+0: -2 [11033.699789] Could not insert probe at lnet_parse+0: -2 [11043.929383] Could not insert probe at lnet_post_send_locked+0: -2 [11043.937052] Could not insert probe at lnet_post_send_locked+0: -2 [11054.192565] Could not insert probe at lnet_parse+0: -2 [11054.201368] Could not insert probe at lnet_parse+0: -2 [11064.441609] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11064.451329] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11074.685133] Could not insert probe at lnet_post_send_locked+0: -2 [11074.693050] Could not insert probe at lnet_post_send_locked+0: -2 [11084.944189] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11084.953540] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11095.191693] Could not insert probe at lnet_parse+0: -2 [11095.201029] Could not insert probe at lnet_parse+0: -2 [11105.442279] Could not insert probe at lnet_send+0: -2 [11105.450446] Could not insert probe at lnet_send+0: -2 [11115.693922] Could not insert probe at lnet_parse+0: -2 [11115.704787] Could not insert probe at lnet_parse+0: -2 [11125.943799] Could not insert probe at lnet_parse+0: -2 [11125.953401] Could not insert probe at lnet_parse+0: -2 [11136.194280] Could not insert probe at lnet_parse+0: -2 [11136.203602] Could not insert probe at lnet_parse+0: -2 [11146.442754] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11146.450970] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11156.699779] Could not insert probe at lnet_parse+0: -2 [11156.710424] Could not insert probe at lnet_parse+0: -2 [11166.944464] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11166.955535] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11177.199263] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11177.210207] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11187.445268] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11187.454403] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11197.700621] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11197.709830] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11207.943271] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11207.954237] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11218.198634] Could not insert probe at lnet_post_send_locked+0: -2 [11218.207433] Could not insert probe at lnet_post_send_locked+0: -2 [11228.447176] Could not insert probe at lnet_parse+0: -2 [11228.455153] Could not insert probe at lnet_parse+0: -2 [11238.698805] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11238.708719] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11248.948790] Could not insert probe at lnet_post_send_locked+0: -2 [11248.957800] Could not insert probe at lnet_post_send_locked+0: -2 [11259.201792] Could not insert probe at lnet_parse+0: -2 [11259.211297] Could not insert probe at lnet_parse+0: -2 [11269.449194] Could not insert probe at lnet_send+0: -2 [11269.458958] Could not insert probe at lnet_send+0: -2 [11279.701488] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11279.710913] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11289.942495] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11289.950385] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11300.203153] Could not insert probe at lnet_send+0: -2 [11300.212840] Could not insert probe at lnet_send+0: -2 [11310.451627] Could not insert probe at lnet_send+0: -2 [11310.461300] Could not insert probe at lnet_send+0: -2 [11320.699281] Could not insert probe at lnet_send+0: -2 [11320.708726] Could not insert probe at lnet_send+0: -2 [11330.947853] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11330.955989] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11341.191687] Could not insert probe at lnet_parse+0: -2 [11341.201504] Could not insert probe at lnet_parse+0: -2 [11351.454477] Could not insert probe at lnet_post_send_locked+0: -2 [11351.463759] Could not insert probe at lnet_post_send_locked+0: -2 [11361.702159] Could not insert probe at lnet_send+0: -2 [11361.711028] Could not insert probe at lnet_send+0: -2 [11371.953080] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11371.962453] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11382.201692] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11382.211256] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11392.459081] Could not insert probe at lnet_send+0: -2 [11392.468315] Could not insert probe at lnet_send+0: -2 [11402.704654] Could not insert probe at lnet_post_send_locked+0: -2 [11402.713731] Could not insert probe at lnet_post_send_locked+0: -2 [11412.944858] Could not insert probe at lnet_parse+0: -2 [11412.952482] Could not insert probe at lnet_parse+0: -2 [11423.196324] Could not insert probe at lnet_send+0: -2 [11423.204322] Could not insert probe at lnet_send+0: -2 [11433.450500] Could not insert probe at lnet_parse+0: -2 [11433.459329] Could not insert probe at lnet_parse+0: -2 [11443.700464] Could not insert probe at lnet_send+0: -2 [11443.709266] Could not insert probe at lnet_send+0: -2 [11453.947667] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11453.955584] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11464.197918] Could not insert probe at lnet_send+0: -2 [11464.206011] Could not insert probe at lnet_send+0: -2 [11474.451365] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11474.461597] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11484.696094] Could not insert probe at lnet_parse+0: -2 [11484.703655] Could not insert probe at lnet_parse+0: -2 [11494.951986] Could not insert probe at lnet_send+0: -2 [11494.961165] Could not insert probe at lnet_send+0: -2 [11505.198719] Could not insert probe at lnet_send+0: -2 [11505.206484] Could not insert probe at lnet_send+0: -2 [11515.452799] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11515.462009] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11525.701319] Could not insert probe at lnet_parse+0: -2 [11525.709070] Could not insert probe at lnet_parse+0: -2 [11535.959569] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11535.967607] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11546.200992] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11546.210204] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11556.457002] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11556.465080] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11566.700526] Could not insert probe at lnet_parse+0: -2 [11566.709935] Could not insert probe at lnet_parse+0: -2 [11576.957979] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11576.966329] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11587.202544] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11587.210386] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11597.457344] Could not insert probe at lnet_parse+0: -2 [11597.465065] Could not insert probe at lnet_parse+0: -2 [11607.712258] Could not insert probe at lnet_parse+0: -2 [11607.720479] Could not insert probe at lnet_parse+0: -2 [11617.957351] Could not insert probe at lnet_send+0: -2 [11617.964896] Could not insert probe at lnet_send+0: -2 [11628.204700] Could not insert probe at lnet_send+0: -2 [11628.212283] Could not insert probe at lnet_send+0: -2 [11638.459156] Could not insert probe at lnet_parse+0: -2 [11638.468082] Could not insert probe at lnet_parse+0: -2 [11648.708623] Could not insert probe at lnet_parse+0: -2 [11648.716002] Could not insert probe at lnet_parse+0: -2 [11658.959326] Could not insert probe at lnet_send+0: -2 [11658.967271] Could not insert probe at lnet_send+0: -2 [11669.207586] Could not insert probe at lnet_post_send_locked+0: -2 [11669.215595] Could not insert probe at lnet_post_send_locked+0: -2 [11679.459183] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11679.469845] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11689.711241] Could not insert probe at lnet_send+0: -2 [11689.718508] Could not insert probe at lnet_send+0: -2 [11699.956378] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11699.965180] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11710.209713] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11710.219322] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11720.465763] Could not insert probe at lnet_parse+0: -2 [11720.473710] Could not insert probe at lnet_parse+0: -2 [11730.706840] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11730.716048] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11740.963911] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11740.973533] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [11751.212633] Could not insert probe at lnet_parse+0: -2 [11751.220473] Could not insert probe at lnet_parse+0: -2 [11761.464108] Could not insert probe at lnet_post_send_locked+0: -2 [11761.473284] Could not insert probe at lnet_post_send_locked+0: -2 [11771.712006] Could not insert probe at lnet_send+0: -2 [11771.719487] Could not insert probe at lnet_send+0: -2 [11781.966367] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11781.975356] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11792.225111] Could not insert probe at lnet_post_send_locked+0: -2 [11792.234294] Could not insert probe at lnet_post_send_locked+0: -2 [11802.465286] Could not insert probe at lnet_parse+0: -2 [11802.472858] Could not insert probe at lnet_parse+0: -2 [11812.713672] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11812.723298] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [11822.961767] Could not insert probe at lnet_post_send_locked+0: -2 [11822.971810] Could not insert probe at lnet_post_send_locked+0: -2 [11833.215846] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11833.223907] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11843.468170] Could not insert probe at lnet_parse+0: -2 [11843.477381] Could not insert probe at lnet_parse+0: -2 [11853.725380] Could not insert probe at lnet_parse+0: -2 [11853.734717] Could not insert probe at lnet_parse+0: -2 [11863.961023] Could not insert probe at lnet_parse+0: -2 [11863.968709] Could not insert probe at lnet_parse+0: -2 [11874.210853] Could not insert probe at lnet_parse+0: -2 [11874.218608] Could not insert probe at lnet_parse+0: -2 [11884.472069] Could not insert probe at lnet_send+0: -2 [11884.482396] Could not insert probe at lnet_send+0: -2 [11894.716703] Could not insert probe at lnet_send+0: -2 [11894.724286] Could not insert probe at lnet_send+0: -2 [11904.977223] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11904.986657] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11915.221169] Could not insert probe at lnet_parse+0: -2 [11915.230978] Could not insert probe at lnet_parse+0: -2 [11925.472205] Could not insert probe at lnet_parse+0: -2 [11925.480493] Could not insert probe at lnet_parse+0: -2 [11935.718494] Could not insert probe at lnet_parse+0: -2 [11935.727834] Could not insert probe at lnet_parse+0: -2 [11945.972253] Could not insert probe at lnet_parse+0: -2 [11945.980219] Could not insert probe at lnet_parse+0: -2 [11956.221177] Could not insert probe at lnet_post_send_locked+0: -2 [11956.229470] Could not insert probe at lnet_post_send_locked+0: -2 [11966.473595] Could not insert probe at lnet_parse+0: -2 [11966.483061] Could not insert probe at lnet_parse+0: -2 [11976.720371] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11976.728289] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [11986.974599] Could not insert probe at lnet_send+0: -2 [11986.982705] Could not insert probe at lnet_send+0: -2 [11997.216953] Could not insert probe at lnet_parse+0: -2 [11997.224688] Could not insert probe at lnet_parse+0: -2 [12007.473467] Could not insert probe at lnet_parse+0: -2 [12007.482637] Could not insert probe at lnet_parse+0: -2 [12017.722602] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12017.731966] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12027.970706] Could not insert probe at lnet_parse+0: -2 [12027.978164] Could not insert probe at lnet_parse+0: -2 [12038.217401] Could not insert probe at lnet_parse+0: -2 [12038.226649] Could not insert probe at lnet_parse+0: -2 [12048.473368] Could not insert probe at lnet_parse+0: -2 [12048.481126] Could not insert probe at lnet_parse+0: -2 [12058.725037] Could not insert probe at lnet_parse+0: -2 [12058.734682] Could not insert probe at lnet_parse+0: -2 [12068.969188] Could not insert probe at lnet_parse+0: -2 [12068.979208] Could not insert probe at lnet_parse+0: -2 [12079.221534] Could not insert probe at lnet_post_send_locked+0: -2 [12079.229206] Could not insert probe at lnet_post_send_locked+0: -2 [12089.473856] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12089.483963] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12099.726113] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12099.733925] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12109.981250] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12109.989459] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12120.223811] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12120.231687] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12130.474545] Could not insert probe at lnet_parse+0: -2 [12130.484075] Could not insert probe at lnet_parse+0: -2 [12140.729037] Could not insert probe at lnet_parse+0: -2 [12140.736481] Could not insert probe at lnet_parse+0: -2 [12150.982690] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12150.991002] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12161.223159] Could not insert probe at lnet_parse+0: -2 [12161.230733] Could not insert probe at lnet_parse+0: -2 [12171.478230] Could not insert probe at lnet_post_send_locked+0: -2 [12171.487716] Could not insert probe at lnet_post_send_locked+0: -2 [12181.732705] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12181.742122] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12191.974981] Could not insert probe at lnet_send+0: -2 [12191.984441] Could not insert probe at lnet_send+0: -2 [12202.225189] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12202.232926] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12212.490388] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12212.500819] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12222.728817] Could not insert probe at lnet_send+0: -2 [12222.736565] Could not insert probe at lnet_send+0: -2 [12232.989640] Could not insert probe at lnet_post_send_locked+0: -2 [12232.999011] Could not insert probe at lnet_post_send_locked+0: -2 [12243.227328] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12243.235422] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12253.478255] Could not insert probe at lnet_parse+0: -2 [12253.485886] Could not insert probe at lnet_parse+0: -2 [12263.728653] Could not insert probe at lnet_parse+0: -2 [12263.738385] Could not insert probe at lnet_parse+0: -2 [12273.983310] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12273.992701] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12284.228370] Could not insert probe at lnet_parse+0: -2 [12284.235927] Could not insert probe at lnet_parse+0: -2 [12294.483324] Could not insert probe at lnet_post_send_locked+0: -2 [12294.492464] Could not insert probe at lnet_post_send_locked+0: -2 [12304.735719] Could not insert probe at lnet_send+0: -2 [12304.743598] Could not insert probe at lnet_send+0: -2 [12314.999174] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12315.008276] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12325.244532] Could not insert probe at lnet_post_send_locked+0: -2 [12325.252652] Could not insert probe at lnet_post_send_locked+0: -2 [12335.486448] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12335.496080] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12345.733163] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12345.741147] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12355.987979] Could not insert probe at lnet_parse+0: -2 [12355.995848] Could not insert probe at lnet_parse+0: -2 [12366.240001] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12366.248125] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12376.491204] Could not insert probe at lnet_parse+0: -2 [12376.499594] Could not insert probe at lnet_parse+0: -2 [12386.738453] Could not insert probe at lnet_post_send_locked+0: -2 [12386.746495] Could not insert probe at lnet_post_send_locked+0: -2 [12396.981390] Could not insert probe at lnet_post_send_locked+0: -2 [12396.990410] Could not insert probe at lnet_post_send_locked+0: -2 [12407.238527] Could not insert probe at lnet_parse+0: -2 [12407.246819] Could not insert probe at lnet_parse+0: -2 [12417.493965] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12417.503231] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12427.741134] Could not insert probe at lnet_send+0: -2 [12427.749798] Could not insert probe at lnet_send+0: -2 [12437.984859] Could not insert probe at lnet_send+0: -2 [12437.992903] Could not insert probe at lnet_send+0: -2 [12448.233724] Could not insert probe at lnet_parse+0: -2 [12448.241294] Could not insert probe at lnet_parse+0: -2 [12458.492459] Could not insert probe at lnet_parse+0: -2 [12458.501773] Could not insert probe at lnet_parse+0: -2 [12468.740243] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12468.750572] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12478.994275] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12479.003212] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12489.239593] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12489.247529] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12499.494264] Could not insert probe at lnet_parse+0: -2 [12499.503322] Could not insert probe at lnet_parse+0: -2 [12509.745467] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12509.753475] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12519.990830] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12519.999039] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12530.237791] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12530.247399] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12540.494005] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12540.501861] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12550.743697] Could not insert probe at lnet_parse+0: -2 [12550.753887] Could not insert probe at lnet_parse+0: -2 [12560.991076] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12560.999284] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12571.243195] Could not insert probe at lnet_post_send_locked+0: -2 [12571.251519] Could not insert probe at lnet_post_send_locked+0: -2 [12581.493833] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12581.503569] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12591.748457] Could not insert probe at lnet_parse+0: -2 [12591.755813] Could not insert probe at lnet_parse+0: -2 [12601.995036] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12602.003251] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12612.248357] Could not insert probe at lnet_parse+0: -2 [12612.256603] Could not insert probe at lnet_parse+0: -2 [12622.498271] Could not insert probe at lnet_parse+0: -2 [12622.505973] Could not insert probe at lnet_parse+0: -2 [12632.750129] Could not insert probe at lnet_parse+0: -2 [12632.758207] Could not insert probe at lnet_parse+0: -2 [12642.993596] Could not insert probe at lnet_post_send_locked+0: -2 [12643.001792] Could not insert probe at lnet_post_send_locked+0: -2 [12653.243799] Could not insert probe at lnet_post_send_locked+0: -2 [12653.251503] Could not insert probe at lnet_post_send_locked+0: -2 [12663.496579] Could not insert probe at lnet_send+0: -2 [12663.504421] Could not insert probe at lnet_send+0: -2 [12673.746878] Could not insert probe at lnet_parse+0: -2 [12673.755855] Could not insert probe at lnet_parse+0: -2 [12683.994406] Could not insert probe at lnet_parse+0: -2 [12684.002235] Could not insert probe at lnet_parse+0: -2 [12694.248566] Could not insert probe at lnet_parse+0: -2 [12694.256349] Could not insert probe at lnet_parse+0: -2 [12704.498215] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12704.508121] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12714.753914] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12714.761918] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12724.997347] Could not insert probe at lnet_parse+0: -2 [12725.004745] Could not insert probe at lnet_parse+0: -2 [12735.254420] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12735.264470] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12745.503875] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12745.514260] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12755.755177] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12755.765810] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12766.005621] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12766.015500] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12776.251934] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12776.260045] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12786.504745] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12786.513218] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12796.759881] Could not insert probe at lnet_parse+0: -2 [12796.769658] Could not insert probe at lnet_parse+0: -2 [12807.003041] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12807.011098] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12817.250849] Could not insert probe at lnet_parse+0: -2 [12817.258262] Could not insert probe at lnet_parse+0: -2 [12827.508951] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12827.518924] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12837.756422] Could not insert probe at lnet_send+0: -2 [12837.766452] Could not insert probe at lnet_send+0: -2 [12848.001922] Could not insert probe at lnet_parse+0: -2 [12848.009495] Could not insert probe at lnet_parse+0: -2 [12858.250458] Could not insert probe at lnet_post_send_locked+0: -2 [12858.259900] Could not insert probe at lnet_post_send_locked+0: -2 [12868.504961] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12868.513355] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12878.758699] Could not insert probe at lnet_send+0: -2 [12878.767684] Could not insert probe at lnet_send+0: -2 [12889.012178] Could not insert probe at lnet_parse+0: -2 [12889.022981] Could not insert probe at lnet_parse+0: -2 [12899.256797] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12899.266563] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [12909.505341] Could not insert probe at lnet_send+0: -2 [12909.512737] Could not insert probe at lnet_send+0: -2 [12919.757601] Could not insert probe at lnet_post_send_locked+0: -2 [12919.767707] Could not insert probe at lnet_post_send_locked+0: -2 [12930.005942] Could not insert probe at lnet_parse+0: -2 [12930.013574] Could not insert probe at lnet_parse+0: -2 [12940.265236] Could not insert probe at lnet_parse+0: -2 [12940.276107] Could not insert probe at lnet_parse+0: -2 [12950.506476] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12950.516773] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [12960.760733] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12960.768769] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12971.007527] Could not insert probe at lnet_post_send_locked+0: -2 [12971.015280] Could not insert probe at lnet_post_send_locked+0: -2 [12981.257657] Could not insert probe at lnet_post_send_locked+0: -2 [12981.266690] Could not insert probe at lnet_post_send_locked+0: -2 [12991.512788] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [12991.520844] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13001.764766] Could not insert probe at lnet_parse+0: -2 [13001.774435] Could not insert probe at lnet_parse+0: -2 [13012.010226] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13012.019054] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13022.259543] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13022.267052] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13032.509545] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13032.517639] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13042.765986] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13042.774021] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13053.015652] Could not insert probe at lnet_post_send_locked+0: -2 [13053.023398] Could not insert probe at lnet_post_send_locked+0: -2 [13063.257872] Could not insert probe at lnet_parse+0: -2 [13063.265594] Could not insert probe at lnet_parse+0: -2 [13073.511667] Could not insert probe at lnet_parse+0: -2 [13073.521191] Could not insert probe at lnet_parse+0: -2 [13083.766019] Could not insert probe at lnet_parse+0: -2 [13083.773591] Could not insert probe at lnet_parse+0: -2 [13094.015727] Could not insert probe at lnet_parse+0: -2 [13094.023988] Could not insert probe at lnet_parse+0: -2 [13104.265343] Could not insert probe at lnet_parse+0: -2 [13104.273371] Could not insert probe at lnet_parse+0: -2 [13114.516240] Could not insert probe at lnet_parse+0: -2 [13114.523766] Could not insert probe at lnet_parse+0: -2 [13124.767536] Could not insert probe at lnet_parse+0: -2 [13124.777139] Could not insert probe at lnet_parse+0: -2 [13135.012892] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13135.021709] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13145.264273] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13145.272215] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13155.514339] Could not insert probe at lnet_parse+0: -2 [13155.523301] Could not insert probe at lnet_parse+0: -2 [13165.771311] Could not insert probe at lnet_parse+0: -2 [13165.778606] Could not insert probe at lnet_parse+0: -2 [13176.018329] Could not insert probe at lnet_parse+0: -2 [13176.026029] Could not insert probe at lnet_parse+0: -2 [13186.267040] Could not insert probe at lnet_parse+0: -2 [13186.274767] Could not insert probe at lnet_parse+0: -2 [13196.520298] Could not insert probe at lnet_post_send_locked+0: -2 [13196.529068] Could not insert probe at lnet_post_send_locked+0: -2 [13206.770536] Could not insert probe at lnet_send+0: -2 [13206.780561] Could not insert probe at lnet_send+0: -2 [13217.023053] Could not insert probe at lnet_send+0: -2 [13217.032628] Could not insert probe at lnet_send+0: -2 [13227.267380] Could not insert probe at lnet_parse+0: -2 [13227.275082] Could not insert probe at lnet_parse+0: -2 [13237.523782] Could not insert probe at lnet_post_send_locked+0: -2 [13237.531321] Could not insert probe at lnet_post_send_locked+0: -2 [13247.775309] Could not insert probe at lnet_parse+0: -2 [13247.782830] Could not insert probe at lnet_parse+0: -2 [13258.019155] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13258.027153] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13268.269082] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13268.276960] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13278.517971] Could not insert probe at lnet_parse+0: -2 [13278.525983] Could not insert probe at lnet_parse+0: -2 [13288.772470] Could not insert probe at lnet_post_send_locked+0: -2 [13288.781541] Could not insert probe at lnet_post_send_locked+0: -2 [13299.018943] Could not insert probe at lnet_parse+0: -2 [13299.026695] Could not insert probe at lnet_parse+0: -2 [13309.271974] Could not insert probe at lnet_parse+0: -2 [13309.279846] Could not insert probe at lnet_parse+0: -2 [13319.519126] Could not insert probe at lnet_parse+0: -2 [13319.527400] Could not insert probe at lnet_parse+0: -2 [13329.775988] Could not insert probe at lnet_post_send_locked+0: -2 [13329.783933] Could not insert probe at lnet_post_send_locked+0: -2 [13340.017914] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13340.025715] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13350.268803] Could not insert probe at lnet_parse+0: -2 [13350.276703] Could not insert probe at lnet_parse+0: -2 [13360.521159] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13360.531036] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13370.776752] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13370.784419] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13381.024681] Could not insert probe at lnet_send+0: -2 [13381.033461] Could not insert probe at lnet_send+0: -2 [13391.275421] Could not insert probe at lnet_send+0: -2 [13391.283219] Could not insert probe at lnet_send+0: -2 [13401.524933] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13401.532703] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13411.778006] Could not insert probe at lnet_parse+0: -2 [13411.786118] Could not insert probe at lnet_parse+0: -2 [13422.022834] Could not insert probe at lnet_parse+0: -2 [13422.030343] Could not insert probe at lnet_parse+0: -2 [13432.272934] Could not insert probe at lnet_send+0: -2 [13432.280768] Could not insert probe at lnet_send+0: -2 [13442.528600] Could not insert probe at lnet_parse+0: -2 [13442.536278] Could not insert probe at lnet_parse+0: -2 [13452.779508] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13452.787527] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13463.031409] Could not insert probe at lnet_send+0: -2 [13463.039045] Could not insert probe at lnet_send+0: -2 [13473.278575] Could not insert probe at lnet_post_send_locked+0: -2 [13473.286461] Could not insert probe at lnet_post_send_locked+0: -2 [13483.526581] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13483.535880] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13493.785490] Could not insert probe at lnet_send+0: -2 [13493.795635] Could not insert probe at lnet_send+0: -2 [13504.029343] Could not insert probe at lnet_post_send_locked+0: -2 [13504.038586] Could not insert probe at lnet_post_send_locked+0: -2 [13514.276362] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13514.285093] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13524.531036] Could not insert probe at lnet_send+0: -2 [13524.540357] Could not insert probe at lnet_send+0: -2 [13534.785826] Could not insert probe at lnet_send+0: -2 [13534.793833] Could not insert probe at lnet_send+0: -2 [13545.027815] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13545.036032] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13555.280749] Could not insert probe at lnet_parse+0: -2 [13555.290663] Could not insert probe at lnet_parse+0: -2 [13565.528171] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13565.535977] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13575.782636] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13575.792366] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13586.035613] Could not insert probe at lnet_parse+0: -2 [13586.043729] Could not insert probe at lnet_parse+0: -2 [13596.287871] Could not insert probe at lnet_parse+0: -2 [13596.296752] Could not insert probe at lnet_parse+0: -2 [13606.537820] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13606.547701] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13616.785521] Could not insert probe at lnet_parse+0: -2 [13616.793697] Could not insert probe at lnet_parse+0: -2 [13627.030448] Could not insert probe at lnet_send+0: -2 [13627.038248] Could not insert probe at lnet_send+0: -2 [13637.292430] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13637.301815] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13647.539096] Could not insert probe at lnet_post_send_locked+0: -2 [13647.549847] Could not insert probe at lnet_post_send_locked+0: -2 [13657.788438] Could not insert probe at lnet_parse+0: -2 [13657.797450] Could not insert probe at lnet_parse+0: -2 [13668.032020] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13668.041620] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13678.295966] Could not insert probe at lnet_parse+0: -2 [13678.305205] Could not insert probe at lnet_parse+0: -2 [13688.541220] Could not insert probe at lnet_post_send_locked+0: -2 [13688.550730] Could not insert probe at lnet_post_send_locked+0: -2 [13698.797649] Could not insert probe at lnet_parse+0: -2 [13698.807314] Could not insert probe at lnet_parse+0: -2 [13709.041431] Could not insert probe at lnet_send+0: -2 [13709.050477] Could not insert probe at lnet_send+0: -2 [13719.297396] Could not insert probe at lnet_post_send_locked+0: -2 [13719.306706] Could not insert probe at lnet_post_send_locked+0: -2 [13729.542325] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13729.552034] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13739.800517] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13739.809752] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13750.041537] Could not insert probe at lnet_send+0: -2 [13750.051140] Could not insert probe at lnet_send+0: -2 [13760.296392] Could not insert probe at lnet_parse+0: -2 [13760.305563] Could not insert probe at lnet_parse+0: -2 [13770.535532] Could not insert probe at lnet_parse+0: -2 [13770.544913] Could not insert probe at lnet_parse+0: -2 [13780.796353] Could not insert probe at lnet_post_send_locked+0: -2 [13780.806463] Could not insert probe at lnet_post_send_locked+0: -2 [13791.044452] Could not insert probe at lnet_parse+0: -2 [13791.053802] Could not insert probe at lnet_parse+0: -2 [13801.299001] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13801.309366] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13811.543834] Could not insert probe at lnet_parse+0: -2 [13811.553482] Could not insert probe at lnet_parse+0: -2 [13821.794904] Could not insert probe at lnet_parse+0: -2 [13821.802913] Could not insert probe at lnet_parse+0: -2 [13832.056298] Could not insert probe at lnet_post_send_locked+0: -2 [13832.064173] Could not insert probe at lnet_post_send_locked+0: -2 [13842.292130] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13842.301565] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13852.540807] Could not insert probe at lnet_post_send_locked+0: -2 [13852.548921] Could not insert probe at lnet_post_send_locked+0: -2 [13862.790886] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13862.800461] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13873.038897] Could not insert probe at lnet_send+0: -2 [13873.047745] Could not insert probe at lnet_send+0: -2 [13883.298256] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13883.306132] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13893.541910] Could not insert probe at lnet_parse+0: -2 [13893.551582] Could not insert probe at lnet_parse+0: -2 [13903.797226] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13903.804969] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13914.040847] Could not insert probe at lnet_parse+0: -2 [13914.049558] Could not insert probe at lnet_parse+0: -2 [13924.291273] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13924.299703] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13934.546596] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13934.555635] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [13944.794125] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13944.803512] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [13955.039781] Could not insert probe at lnet_post_send_locked+0: -2 [13955.047566] Could not insert probe at lnet_post_send_locked+0: -2 [13965.294373] Could not insert probe at lnet_post_send_locked+0: -2 [13965.302109] Could not insert probe at lnet_post_send_locked+0: -2 [13975.544516] Could not insert probe at lnet_parse+0: -2 [13975.552178] Could not insert probe at lnet_parse+0: -2 [13985.796813] Could not insert probe at lnet_post_send_locked+0: -2 [13985.806112] Could not insert probe at lnet_post_send_locked+0: -2 [13996.046462] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [13996.054302] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14006.300716] Could not insert probe at lnet_parse+0: -2 [14006.308861] Could not insert probe at lnet_parse+0: -2 [14016.551420] Could not insert probe at lnet_parse+0: -2 [14016.561258] Could not insert probe at lnet_parse+0: -2 [14026.800955] Could not insert probe at lnet_post_send_locked+0: -2 [14026.808649] Could not insert probe at lnet_post_send_locked+0: -2 [14037.048298] Could not insert probe at lnet_post_send_locked+0: -2 [14037.055807] Could not insert probe at lnet_post_send_locked+0: -2 [14047.296204] Could not insert probe at lnet_parse+0: -2 [14047.305335] Could not insert probe at lnet_parse+0: -2 [14057.549292] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14057.557289] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14067.800877] Could not insert probe at lnet_post_send_locked+0: -2 [14067.810368] Could not insert probe at lnet_post_send_locked+0: -2 [14078.050453] Could not insert probe at lnet_parse+0: -2 [14078.058534] Could not insert probe at lnet_parse+0: -2 [14088.298334] Could not insert probe at lnet_parse+0: -2 [14088.305815] Could not insert probe at lnet_parse+0: -2 [14098.549205] Could not insert probe at lnet_post_send_locked+0: -2 [14098.557009] Could not insert probe at lnet_post_send_locked+0: -2 [14108.803390] Could not insert probe at lnet_post_send_locked+0: -2 [14108.812799] Could not insert probe at lnet_post_send_locked+0: -2 [14119.049063] Could not insert probe at lnet_post_send_locked+0: -2 [14119.056981] Could not insert probe at lnet_post_send_locked+0: -2 [14129.303849] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14129.311551] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14139.551400] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14139.560994] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14149.799782] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14149.807818] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14160.050087] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14160.058625] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14170.304279] Could not insert probe at lnet_parse+0: -2 [14170.313036] Could not insert probe at lnet_parse+0: -2 [14180.552987] Could not insert probe at lnet_send+0: -2 [14180.560501] Could not insert probe at lnet_send+0: -2 [14190.805635] Could not insert probe at lnet_send+0: -2 [14190.815324] Could not insert probe at lnet_send+0: -2 [14201.054553] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14201.062327] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14211.308398] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14211.316734] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14221.560164] Could not insert probe at lnet_post_send_locked+0: -2 [14221.568105] Could not insert probe at lnet_post_send_locked+0: -2 [14231.801926] Could not insert probe at lnet_parse+0: -2 [14231.811949] Could not insert probe at lnet_parse+0: -2 [14242.054910] Could not insert probe at lnet_post_send_locked+0: -2 [14242.062842] Could not insert probe at lnet_post_send_locked+0: -2 [14252.310269] Could not insert probe at lnet_send+0: -2 [14252.318306] Could not insert probe at lnet_send+0: -2 [14262.554895] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14262.565219] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14272.813055] Could not insert probe at lnet_parse+0: -2 [14272.820952] Could not insert probe at lnet_parse+0: -2 [14283.061005] Could not insert probe at lnet_parse+0: -2 [14283.069277] Could not insert probe at lnet_parse+0: -2 [14293.305050] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14293.312753] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14303.561166] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14303.569263] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14313.807439] Could not insert probe at lnet_post_send_locked+0: -2 [14313.816635] Could not insert probe at lnet_post_send_locked+0: -2 [14324.057876] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14324.065771] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14334.306497] Could not insert probe at lnet_parse+0: -2 [14334.313714] Could not insert probe at lnet_parse+0: -2 [14344.558344] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14344.566189] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14354.810190] Could not insert probe at lnet_parse+0: -2 [14354.818247] Could not insert probe at lnet_parse+0: -2 [14365.063665] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14365.071540] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14375.311763] Could not insert probe at lnet_send+0: -2 [14375.319991] Could not insert probe at lnet_send+0: -2 [14385.561510] Could not insert probe at lnet_post_send_locked+0: -2 [14385.569347] Could not insert probe at lnet_post_send_locked+0: -2 [14395.813226] Could not insert probe at lnet_parse+0: -2 [14395.820764] Could not insert probe at lnet_parse+0: -2 [14406.064420] Could not insert probe at lnet_send+0: -2 [14406.072146] Could not insert probe at lnet_send+0: -2 [14416.315066] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14416.323939] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14426.564029] Could not insert probe at lnet_post_send_locked+0: -2 [14426.571989] Could not insert probe at lnet_post_send_locked+0: -2 [14436.814441] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14436.824734] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14447.065337] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14447.073257] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14457.319982] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14457.328185] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14467.568395] Could not insert probe at lnet_send+0: -2 [14467.577559] Could not insert probe at lnet_send+0: -2 [14477.814766] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14477.822628] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14488.067765] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14488.076777] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14498.316648] Could not insert probe at lnet_send+0: -2 [14498.324487] Could not insert probe at lnet_send+0: -2 [14508.568543] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14508.576621] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14518.818625] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14518.828873] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14529.065124] Could not insert probe at lnet_send+0: -2 [14529.072717] Could not insert probe at lnet_send+0: -2 [14539.314218] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14539.322222] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14549.576298] Could not insert probe at lnet_parse+0: -2 [14549.585663] Could not insert probe at lnet_parse+0: -2 [14559.831658] Could not insert probe at lnet_parse+0: -2 [14559.841283] Could not insert probe at lnet_parse+0: -2 [14570.075873] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14570.084340] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14580.317984] Could not insert probe at lnet_parse+0: -2 [14580.325752] Could not insert probe at lnet_parse+0: -2 [14590.571447] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14590.579269] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14600.818848] Could not insert probe at lnet_post_send_locked+0: -2 [14600.826595] Could not insert probe at lnet_post_send_locked+0: -2 [14611.075270] Could not insert probe at lnet_post_send_locked+0: -2 [14611.085095] Could not insert probe at lnet_post_send_locked+0: -2 [14621.326487] Could not insert probe at lnet_parse+0: -2 [14621.334483] Could not insert probe at lnet_parse+0: -2 [14631.571686] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14631.579867] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14641.823417] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14641.833197] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14652.072386] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14652.080577] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14662.320662] Could not insert probe at lnet_parse+0: -2 [14662.329109] Could not insert probe at lnet_parse+0: -2 [14672.577268] Could not insert probe at lnet_post_send_locked+0: -2 [14672.585232] Could not insert probe at lnet_post_send_locked+0: -2 [14682.823048] Could not insert probe at lnet_parse+0: -2 [14682.830705] Could not insert probe at lnet_parse+0: -2 [14693.073652] Could not insert probe at lnet_post_send_locked+0: -2 [14693.081147] Could not insert probe at lnet_post_send_locked+0: -2 [14703.325772] Could not insert probe at lnet_parse+0: -2 [14703.333564] Could not insert probe at lnet_parse+0: -2 [14713.574260] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14713.581922] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14723.827075] Could not insert probe at lnet_post_send_locked+0: -2 [14723.835292] Could not insert probe at lnet_post_send_locked+0: -2 [14734.075169] Could not insert probe at lnet_parse+0: -2 [14734.082657] Could not insert probe at lnet_parse+0: -2 [14744.328099] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14744.336171] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14754.573425] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14754.581548] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14764.824255] Could not insert probe at lnet_post_send_locked+0: -2 [14764.833926] Could not insert probe at lnet_post_send_locked+0: -2 [14775.076605] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14775.084522] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14785.340010] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14785.347749] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14795.574707] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14795.584079] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14805.823994] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14805.832770] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14816.081167] Could not insert probe at lnet_send+0: -2 [14816.089973] Could not insert probe at lnet_send+0: -2 [14826.334043] Could not insert probe at lnet_post_send_locked+0: -2 [14826.343392] Could not insert probe at lnet_post_send_locked+0: -2 [14836.582615] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14836.591810] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14846.829584] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14846.838165] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14857.083551] Could not insert probe at lnet_parse+0: -2 [14857.092543] Could not insert probe at lnet_parse+0: -2 [14867.327787] Could not insert probe at lnet_send+0: -2 [14867.335647] Could not insert probe at lnet_send+0: -2 [14877.578563] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14877.588349] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14887.836851] Could not insert probe at lnet_post_send_locked+0: -2 [14887.846771] Could not insert probe at lnet_post_send_locked+0: -2 [14898.083628] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14898.091630] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [14908.337591] Could not insert probe at lnet_post_send_locked+0: -2 [14908.345474] Could not insert probe at lnet_post_send_locked+0: -2 [14918.589006] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14918.599432] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14928.829067] Could not insert probe at lnet_post_send_locked+0: -2 [14928.837175] Could not insert probe at lnet_post_send_locked+0: -2 [14939.084110] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14939.093633] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14949.330514] Could not insert probe at lnet_post_send_locked+0: -2 [14949.339362] Could not insert probe at lnet_post_send_locked+0: -2 [14959.591033] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14959.599475] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [14969.831293] Could not insert probe at lnet_parse+0: -2 [14969.838718] Could not insert probe at lnet_parse+0: -2 [14980.086682] Could not insert probe at lnet_parse+0: -2 [14980.095159] Could not insert probe at lnet_parse+0: -2 [14990.332524] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [14990.340592] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15000.585466] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15000.594496] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15010.846585] Could not insert probe at lnet_parse+0: -2 [15010.857407] Could not insert probe at lnet_parse+0: -2 [15021.089633] Could not insert probe at lnet_post_send_locked+0: -2 [15021.098825] Could not insert probe at lnet_post_send_locked+0: -2 [15031.339146] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15031.347227] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15041.592666] Could not insert probe at lnet_send+0: -2 [15041.603509] Could not insert probe at lnet_send+0: -2 [15051.839607] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15051.847977] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15062.089606] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15062.099331] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15072.337013] Could not insert probe at lnet_parse+0: -2 [15072.344824] Could not insert probe at lnet_parse+0: -2 [15082.587624] Could not insert probe at lnet_send+0: -2 [15082.597123] Could not insert probe at lnet_send+0: -2 [15092.838941] Could not insert probe at lnet_parse+0: -2 [15092.846575] Could not insert probe at lnet_parse+0: -2 [15103.097272] Could not insert probe at lnet_parse+0: -2 [15103.106687] Could not insert probe at lnet_parse+0: -2 [15113.338628] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15113.346411] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15123.588455] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15123.597829] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15133.842412] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15133.850090] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15144.094049] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15144.103488] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15154.339088] Could not insert probe at lnet_parse+0: -2 [15154.346896] Could not insert probe at lnet_parse+0: -2 [15164.590851] Could not insert probe at lnet_parse+0: -2 [15164.599165] Could not insert probe at lnet_parse+0: -2 [15174.843507] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15174.852781] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15185.096334] Could not insert probe at lnet_parse+0: -2 [15185.103911] Could not insert probe at lnet_parse+0: -2 [15195.348251] Could not insert probe at lnet_parse+0: -2 [15195.357443] Could not insert probe at lnet_parse+0: -2 [15205.602115] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15205.609998] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15215.849239] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15215.860188] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15226.104497] Could not insert probe at lnet_parse+0: -2 [15226.113891] Could not insert probe at lnet_parse+0: -2 [15236.350860] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15236.360558] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15246.605088] Could not insert probe at lnet_parse+0: -2 [15246.614842] Could not insert probe at lnet_parse+0: -2 [15256.847392] Could not insert probe at lnet_parse+0: -2 [15256.856828] Could not insert probe at lnet_parse+0: -2 [15267.105470] Could not insert probe at lnet_post_send_locked+0: -2 [15267.115025] Could not insert probe at lnet_post_send_locked+0: -2 [15277.351034] Could not insert probe at lnet_post_send_locked+0: -2 [15277.360646] Could not insert probe at lnet_post_send_locked+0: -2 [15287.598088] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15287.608609] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15297.845589] Could not insert probe at lnet_parse+0: -2 [15297.854562] Could not insert probe at lnet_parse+0: -2 [15308.104329] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15308.113837] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15318.354481] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15318.363998] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15328.596008] Could not insert probe at lnet_parse+0: -2 [15328.603611] Could not insert probe at lnet_parse+0: -2 [15338.858189] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15338.869088] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15349.104226] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15349.113425] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15359.350014] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15359.357819] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15369.606172] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15369.617154] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15379.861951] Could not insert probe at lnet_post_send_locked+0: -2 [15379.869778] Could not insert probe at lnet_post_send_locked+0: -2 [15390.107520] Could not insert probe at lnet_parse+0: -2 [15390.116763] Could not insert probe at lnet_parse+0: -2 [15400.362628] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15400.372354] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15410.608277] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15410.617523] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15420.865393] Could not insert probe at lnet_parse+0: -2 [15420.875637] Could not insert probe at lnet_parse+0: -2 [15431.106483] Could not insert probe at lnet_parse+0: -2 [15431.115584] Could not insert probe at lnet_parse+0: -2 [15441.362904] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15441.373956] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15451.609149] Could not insert probe at lnet_parse+0: -2 [15451.618185] Could not insert probe at lnet_parse+0: -2 [15461.863751] Could not insert probe at lnet_post_send_locked+0: -2 [15461.873782] Could not insert probe at lnet_post_send_locked+0: -2 [15472.110069] Could not insert probe at lnet_parse+0: -2 [15472.119140] Could not insert probe at lnet_parse+0: -2 [15482.358660] Could not insert probe at lnet_send+0: -2 [15482.366525] Could not insert probe at lnet_send+0: -2 [15492.612441] Could not insert probe at lnet_parse+0: -2 [15492.622027] Could not insert probe at lnet_parse+0: -2 [15502.863476] Could not insert probe at lnet_send+0: -2 [15502.873734] Could not insert probe at lnet_send+0: -2 [15513.116087] Could not insert probe at lnet_send+0: -2 [15513.125643] Could not insert probe at lnet_send+0: -2 [15523.361759] Could not insert probe at lnet_send+0: -2 [15523.371088] Could not insert probe at lnet_send+0: -2 [15533.612724] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15533.622890] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15543.862072] Could not insert probe at lnet_send+0: -2 [15543.870930] Could not insert probe at lnet_send+0: -2 [15554.118315] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15554.126322] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15564.362559] Could not insert probe at lnet_parse+0: -2 [15564.371633] Could not insert probe at lnet_parse+0: -2 [15574.605631] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15574.613642] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15584.864131] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15584.873311] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15595.118892] Could not insert probe at lnet_post_send_locked+0: -2 [15595.127649] Could not insert probe at lnet_post_send_locked+0: -2 [15605.361603] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15605.371224] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15615.612856] Could not insert probe at lnet_parse+0: -2 [15615.620729] Could not insert probe at lnet_parse+0: -2 [15625.860523] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15625.870129] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15636.113327] Could not insert probe at lnet_parse+0: -2 [15636.121213] Could not insert probe at lnet_parse+0: -2 [15646.365842] Could not insert probe at lnet_send+0: -2 [15646.373941] Could not insert probe at lnet_send+0: -2 [15656.612447] Could not insert probe at lnet_parse+0: -2 [15656.621358] Could not insert probe at lnet_parse+0: -2 [15666.859418] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15666.867622] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15677.113945] Could not insert probe at lnet_post_send_locked+0: -2 [15677.121696] Could not insert probe at lnet_post_send_locked+0: -2 [15687.367318] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15687.376761] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15697.618372] Could not insert probe at lnet_parse+0: -2 [15697.628012] Could not insert probe at lnet_parse+0: -2 [15707.860363] Could not insert probe at lnet_send+0: -2 [15707.869177] Could not insert probe at lnet_send+0: -2 [15718.113809] Could not insert probe at lnet_post_send_locked+0: -2 [15718.121702] Could not insert probe at lnet_post_send_locked+0: -2 [15728.359878] Could not insert probe at lnet_send+0: -2 [15728.368115] Could not insert probe at lnet_send+0: -2 [15738.621410] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15738.632863] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15748.863263] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15748.872766] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15759.113871] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15759.122343] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15769.362642] Could not insert probe at lnet_post_send_locked+0: -2 [15769.370576] Could not insert probe at lnet_post_send_locked+0: -2 [15779.618084] Could not insert probe at lnet_post_send_locked+0: -2 [15779.626092] Could not insert probe at lnet_post_send_locked+0: -2 [15789.864158] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15789.872450] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15800.122044] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15800.131951] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15810.362751] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15810.371294] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15820.614944] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15820.624633] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15830.869647] Could not insert probe at lnet_post_send_locked+0: -2 [15830.878635] Could not insert probe at lnet_post_send_locked+0: -2 [15841.121534] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15841.130559] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15851.370060] Could not insert probe at lnet_parse+0: -2 [15851.377690] Could not insert probe at lnet_parse+0: -2 [15861.619648] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15861.628585] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15871.881355] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15871.892609] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15882.120847] Could not insert probe at lnet_send+0: -2 [15882.130155] Could not insert probe at lnet_send+0: -2 [15892.368539] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15892.376372] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [15902.620151] Could not insert probe at lnet_post_send_locked+0: -2 [15902.627842] Could not insert probe at lnet_post_send_locked+0: -2 [15912.873200] Could not insert probe at lnet_parse+0: -2 [15912.882266] Could not insert probe at lnet_parse+0: -2 [15923.125647] Could not insert probe at lnet_parse+0: -2 [15923.133613] Could not insert probe at lnet_parse+0: -2 [15933.373457] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15933.382517] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [15943.619160] Could not insert probe at lnet_parse+0: -2 [15943.626997] Could not insert probe at lnet_parse+0: -2 [15953.876987] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15953.884691] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [15964.129739] Could not insert probe at lnet_parse+0: -2 [15964.139320] Could not insert probe at lnet_parse+0: -2 [15974.371809] Could not insert probe at lnet_send+0: -2 [15974.379601] Could not insert probe at lnet_send+0: -2 [15984.621885] Could not insert probe at lnet_post_send_locked+0: -2 [15984.629993] Could not insert probe at lnet_post_send_locked+0: -2 [15994.870425] Could not insert probe at lnet_parse+0: -2 [15994.879323] Could not insert probe at lnet_parse+0: -2 [16005.128270] Could not insert probe at lnet_send+0: -2 [16005.135938] Could not insert probe at lnet_send+0: -2 [16015.381898] Could not insert probe at lnet_post_send_locked+0: -2 [16015.391274] Could not insert probe at lnet_post_send_locked+0: -2 [16025.627271] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16025.635581] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16035.879527] Could not insert probe at lnet_post_send_locked+0: -2 [16035.888879] Could not insert probe at lnet_post_send_locked+0: -2 [16046.130791] Could not insert probe at lnet_parse+0: -2 [16046.140561] Could not insert probe at lnet_parse+0: -2 [16056.375340] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16056.383464] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16066.631968] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16066.640370] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16076.877536] Could not insert probe at lnet_parse+0: -2 [16076.886620] Could not insert probe at lnet_parse+0: -2 [16087.124696] Could not insert probe at lnet_parse+0: -2 [16087.132551] Could not insert probe at lnet_parse+0: -2 [16097.378734] Could not insert probe at lnet_parse+0: -2 [16097.386160] Could not insert probe at lnet_parse+0: -2 [16107.629127] Could not insert probe at lnet_parse+0: -2 [16107.639350] Could not insert probe at lnet_parse+0: -2 [16117.875166] Could not insert probe at lnet_parse+0: -2 [16117.883089] Could not insert probe at lnet_parse+0: -2 [16128.129164] Could not insert probe at lnet_parse+0: -2 [16128.138571] Could not insert probe at lnet_parse+0: -2 [16138.377063] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16138.386843] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16148.635985] Could not insert probe at lnet_parse+0: -2 [16148.645087] Could not insert probe at lnet_parse+0: -2 [16158.879106] Could not insert probe at lnet_parse+0: -2 [16158.886527] Could not insert probe at lnet_parse+0: -2 [16169.133099] Could not insert probe at lnet_post_send_locked+0: -2 [16169.142123] Could not insert probe at lnet_post_send_locked+0: -2 [16179.387904] Could not insert probe at lnet_send+0: -2 [16179.397456] Could not insert probe at lnet_send+0: -2 [16189.636146] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16189.645715] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16199.877426] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16199.887734] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16210.126781] Could not insert probe at lnet_parse+0: -2 [16210.134730] Could not insert probe at lnet_parse+0: -2 [16220.389313] Could not insert probe at lnet_send+0: -2 [16220.400068] Could not insert probe at lnet_send+0: -2 [16230.633042] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16230.641508] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16240.886705] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16240.896944] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16251.128770] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16251.136871] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16261.389337] Could not insert probe at lnet_parse+0: -2 [16261.397782] Could not insert probe at lnet_parse+0: -2 [16271.640473] Could not insert probe at lnet_parse+0: -2 [16271.650083] Could not insert probe at lnet_parse+0: -2 [16281.891953] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16281.901617] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16292.138828] Could not insert probe at lnet_parse+0: -2 [16292.147877] Could not insert probe at lnet_parse+0: -2 [16302.382448] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16302.390251] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16312.642763] Could not insert probe at lnet_post_send_locked+0: -2 [16312.653674] Could not insert probe at lnet_post_send_locked+0: -2 [16322.885769] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16322.894944] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16333.134042] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16333.142326] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16343.390395] Could not insert probe at lnet_parse+0: -2 [16343.398809] Could not insert probe at lnet_parse+0: -2 [16353.637276] Could not insert probe at lnet_send+0: -2 [16353.647266] Could not insert probe at lnet_send+0: -2 [16363.884179] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16363.892610] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16374.135661] Could not insert probe at lnet_parse+0: -2 [16374.144986] Could not insert probe at lnet_parse+0: -2 [16384.386345] Could not insert probe at lnet_parse+0: -2 [16384.394108] Could not insert probe at lnet_parse+0: -2 [16394.641113] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16394.648838] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16404.886706] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16404.894742] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16415.135536] Could not insert probe at lnet_send+0: -2 [16415.144628] Could not insert probe at lnet_send+0: -2 [16425.392475] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16425.401887] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16435.639094] Could not insert probe at lnet_parse+0: -2 [16435.646862] Could not insert probe at lnet_parse+0: -2 [16445.888320] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16445.897751] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16456.138198] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16456.146153] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16466.396771] Could not insert probe at lnet_parse+0: -2 [16466.404554] Could not insert probe at lnet_parse+0: -2 [16476.643168] Could not insert probe at lnet_send+0: -2 [16476.650946] Could not insert probe at lnet_send+0: -2 [16486.890670] Could not insert probe at lnet_send+0: -2 [16486.898325] Could not insert probe at lnet_send+0: -2 [16497.157464] Could not insert probe at lnet_parse+0: -2 [16497.166982] Could not insert probe at lnet_parse+0: -2 [16507.395055] Could not insert probe at lnet_send+0: -2 [16507.403154] Could not insert probe at lnet_send+0: -2 [16517.647075] Could not insert probe at lnet_parse+0: -2 [16517.655011] Could not insert probe at lnet_parse+0: -2 [16527.895756] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16527.904004] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16538.154864] Could not insert probe at lnet_parse+0: -2 [16538.164226] Could not insert probe at lnet_parse+0: -2 [16548.397134] Could not insert probe at lnet_parse+0: -2 [16548.404630] Could not insert probe at lnet_parse+0: -2 [16558.647639] Could not insert probe at lnet_post_send_locked+0: -2 [16558.655662] Could not insert probe at lnet_post_send_locked+0: -2 [16568.896017] Could not insert probe at lnet_send+0: -2 [16568.905562] Could not insert probe at lnet_send+0: -2 [16579.153145] Could not insert probe at lnet_parse+0: -2 [16579.163170] Could not insert probe at lnet_parse+0: -2 [16589.396140] Could not insert probe at lnet_parse+0: -2 [16589.404150] Could not insert probe at lnet_parse+0: -2 [16599.652598] Could not insert probe at lnet_post_send_locked+0: -2 [16599.661638] Could not insert probe at lnet_post_send_locked+0: -2 [16609.905699] Could not insert probe at lnet_post_send_locked+0: -2 [16609.915276] Could not insert probe at lnet_post_send_locked+0: -2 [16620.152643] Could not insert probe at lnet_parse+0: -2 [16620.160565] Could not insert probe at lnet_parse+0: -2 [16630.401734] Could not insert probe at lnet_parse+0: -2 [16630.409769] Could not insert probe at lnet_parse+0: -2 [16640.646084] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16640.654421] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16650.899183] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16650.907669] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16661.150836] Could not insert probe at lnet_post_send_locked+0: -2 [16661.160157] Could not insert probe at lnet_post_send_locked+0: -2 [16671.406944] Could not insert probe at lnet_post_send_locked+0: -2 [16671.414837] Could not insert probe at lnet_post_send_locked+0: -2 [16681.656790] Could not insert probe at lnet_send+0: -2 [16681.665879] Could not insert probe at lnet_send+0: -2 [16691.902356] Could not insert probe at lnet_parse+0: -2 [16691.910234] Could not insert probe at lnet_parse+0: -2 [16702.151497] Could not insert probe at lnet_send+0: -2 [16702.160693] Could not insert probe at lnet_send+0: -2 [16712.403557] Could not insert probe at lnet_send+0: -2 [16712.411141] Could not insert probe at lnet_send+0: -2 [16722.647840] Could not insert probe at lnet_post_send_locked+0: -2 [16722.657491] Could not insert probe at lnet_post_send_locked+0: -2 [16732.905310] Could not insert probe at lnet_parse+0: -2 [16732.912778] Could not insert probe at lnet_parse+0: -2 [16743.156441] Could not insert probe at lnet_post_send_locked+0: -2 [16743.165214] Could not insert probe at lnet_post_send_locked+0: -2 [16753.407497] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16753.416618] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16763.664739] Could not insert probe at lnet_parse+0: -2 [16763.674379] Could not insert probe at lnet_parse+0: -2 [16773.903882] Could not insert probe at lnet_post_send_locked+0: -2 [16773.912770] Could not insert probe at lnet_post_send_locked+0: -2 [16784.150445] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16784.159754] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16794.404507] Could not insert probe at lnet_post_send_locked+0: -2 [16794.412354] Could not insert probe at lnet_post_send_locked+0: -2 [16804.661862] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16804.671814] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16814.908004] Could not insert probe at lnet_parse+0: -2 [16814.916143] Could not insert probe at lnet_parse+0: -2 [16825.157114] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16825.166527] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16835.411115] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16835.418902] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16845.654424] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16845.662376] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16855.909392] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16855.918807] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [16866.154878] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16866.164042] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16876.414522] Could not insert probe at lnet_parse+0: -2 [16876.422341] Could not insert probe at lnet_parse+0: -2 [16886.670809] Could not insert probe at lnet_send+0: -2 [16886.680053] Could not insert probe at lnet_send+0: -2 [16896.905086] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16896.913035] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16907.163252] Could not insert probe at lnet_send+0: -2 [16907.172735] Could not insert probe at lnet_send+0: -2 [16917.411121] Could not insert probe at lnet_post_send_locked+0: -2 [16917.419025] Could not insert probe at lnet_post_send_locked+0: -2 [16927.658354] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16927.666586] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16937.912730] Could not insert probe at lnet_parse+0: -2 [16937.921905] Could not insert probe at lnet_parse+0: -2 [16948.157099] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16948.165010] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [16958.413880] Could not insert probe at lnet_parse+0: -2 [16958.421761] Could not insert probe at lnet_parse+0: -2 [16968.660055] Could not insert probe at lnet_parse+0: -2 [16968.668134] Could not insert probe at lnet_parse+0: -2 [16978.913975] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16978.923541] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [16989.158013] Could not insert probe at lnet_parse+0: -2 [16989.165570] Could not insert probe at lnet_parse+0: -2 [16999.415836] Could not insert probe at lnet_send+0: -2 [16999.424753] Could not insert probe at lnet_send+0: -2 [17009.660760] Could not insert probe at lnet_send+0: -2 [17009.669789] Could not insert probe at lnet_send+0: -2 [17019.912891] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17019.920934] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17030.161125] Could not insert probe at lnet_parse+0: -2 [17030.170449] Could not insert probe at lnet_parse+0: -2 [17040.411121] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17040.419016] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17050.661557] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17050.669389] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17060.916396] Could not insert probe at lnet_post_send_locked+0: -2 [17060.925748] Could not insert probe at lnet_post_send_locked+0: -2 [17071.172575] Could not insert probe at lnet_parse+0: -2 [17071.181906] Could not insert probe at lnet_parse+0: -2 [17081.425092] Could not insert probe at lnet_parse+0: -2 [17081.434830] Could not insert probe at lnet_parse+0: -2 [17091.668327] Could not insert probe at lnet_parse+0: -2 [17091.675969] Could not insert probe at lnet_parse+0: -2 [17101.928180] Could not insert probe at lnet_parse+0: -2 [17101.936020] Could not insert probe at lnet_parse+0: -2 [17112.170447] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17112.178233] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17122.414310] Could not insert probe at lnet_parse+0: -2 [17122.422011] Could not insert probe at lnet_parse+0: -2 [17132.665693] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17132.673502] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17142.916459] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17142.925270] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17153.168239] Could not insert probe at lnet_parse+0: -2 [17153.176198] Could not insert probe at lnet_parse+0: -2 [17163.426229] Could not insert probe at lnet_send+0: -2 [17163.433721] Could not insert probe at lnet_send+0: -2 [17173.668362] Could not insert probe at lnet_parse+0: -2 [17173.676155] Could not insert probe at lnet_parse+0: -2 [17183.923593] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17183.931498] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17194.165858] Could not insert probe at lnet_parse+0: -2 [17194.173850] Could not insert probe at lnet_parse+0: -2 [17204.423686] Could not insert probe at lnet_parse+0: -2 [17204.431288] Could not insert probe at lnet_parse+0: -2 [17214.670015] Could not insert probe at lnet_parse+0: -2 [17214.677741] Could not insert probe at lnet_parse+0: -2 [17224.929293] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17224.939469] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17235.171869] Could not insert probe at lnet_parse+0: -2 [17235.181892] Could not insert probe at lnet_parse+0: -2 [17245.434784] Could not insert probe at lnet_post_send_locked+0: -2 [17245.446402] Could not insert probe at lnet_post_send_locked+0: -2 [17255.673127] Could not insert probe at lnet_parse+0: -2 [17255.682428] Could not insert probe at lnet_parse+0: -2 [17265.921890] Could not insert probe at lnet_send+0: -2 [17265.929727] Could not insert probe at lnet_send+0: -2 [17276.170683] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17276.179982] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17286.426181] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17286.435592] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17296.677381] Could not insert probe at lnet_post_send_locked+0: -2 [17296.686324] Could not insert probe at lnet_post_send_locked+0: -2 [17306.923007] Could not insert probe at lnet_parse+0: -2 [17306.931295] Could not insert probe at lnet_parse+0: -2 [17317.171339] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17317.179161] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17327.423828] Could not insert probe at lnet_parse+0: -2 [17327.432052] Could not insert probe at lnet_parse+0: -2 [17337.680764] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17337.688608] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17347.936021] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17347.943892] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17358.174302] Could not insert probe at lnet_parse+0: -2 [17358.182037] Could not insert probe at lnet_parse+0: -2 [17368.424567] Could not insert probe at lnet_send+0: -2 [17368.432655] Could not insert probe at lnet_send+0: -2 [17378.679720] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17378.688027] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17388.929095] Could not insert probe at lnet_parse+0: -2 [17388.938558] Could not insert probe at lnet_parse+0: -2 [17399.176534] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17399.184326] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17409.426387] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17409.434274] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17419.680514] Could not insert probe at lnet_post_send_locked+0: -2 [17419.688789] Could not insert probe at lnet_post_send_locked+0: -2 [17429.928595] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17429.937317] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17440.177610] Could not insert probe at lnet_parse+0: -2 [17440.186521] Could not insert probe at lnet_parse+0: -2 [17450.426040] Could not insert probe at lnet_post_send_locked+0: -2 [17450.433857] Could not insert probe at lnet_post_send_locked+0: -2 [17460.683214] Could not insert probe at lnet_parse+0: -2 [17460.692487] Could not insert probe at lnet_parse+0: -2 [17470.928229] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17470.936043] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17481.182266] Could not insert probe at lnet_parse+0: -2 [17481.191133] Could not insert probe at lnet_parse+0: -2 [17491.435232] Could not insert probe at lnet_post_send_locked+0: -2 [17491.443652] Could not insert probe at lnet_post_send_locked+0: -2 [17501.687950] Could not insert probe at lnet_parse+0: -2 [17501.695999] Could not insert probe at lnet_parse+0: -2 [17511.938014] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17511.946318] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17522.190554] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17522.199772] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17532.430964] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17532.439039] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17542.688618] Could not insert probe at lnet_parse+0: -2 [17542.697600] Could not insert probe at lnet_parse+0: -2 [17552.931336] Could not insert probe at lnet_parse+0: -2 [17552.938774] Could not insert probe at lnet_parse+0: -2 [17563.179947] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17563.189229] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17573.433711] Could not insert probe at lnet_post_send_locked+0: -2 [17573.441766] Could not insert probe at lnet_post_send_locked+0: -2 [17583.683468] Could not insert probe at lnet_send+0: -2 [17583.691188] Could not insert probe at lnet_send+0: -2 [17593.935092] Could not insert probe at lnet_parse+0: -2 [17593.944436] Could not insert probe at lnet_parse+0: -2 [17604.183934] Could not insert probe at lnet_parse+0: -2 [17604.191803] Could not insert probe at lnet_parse+0: -2 [17614.434354] Could not insert probe at lnet_send+0: -2 [17614.442290] Could not insert probe at lnet_send+0: -2 [17624.687135] Could not insert probe at lnet_parse+0: -2 [17624.694803] Could not insert probe at lnet_parse+0: -2 [17634.941625] Could not insert probe at lnet_parse+0: -2 [17634.951227] Could not insert probe at lnet_parse+0: -2 [17645.191271] Could not insert probe at lnet_parse+0: -2 [17645.199558] Could not insert probe at lnet_parse+0: -2 [17655.435313] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17655.444309] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17665.690505] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17665.700407] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17675.939756] Could not insert probe at lnet_post_send_locked+0: -2 [17675.947655] Could not insert probe at lnet_post_send_locked+0: -2 [17686.187113] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17686.196128] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [17696.441325] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17696.449431] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17706.696643] Could not insert probe at lnet_post_send_locked+0: -2 [17706.705054] Could not insert probe at lnet_post_send_locked+0: -2 [17716.939423] Could not insert probe at lnet_post_send_locked+0: -2 [17716.948396] Could not insert probe at lnet_post_send_locked+0: -2 [17727.187586] Could not insert probe at lnet_parse+0: -2 [17727.195673] Could not insert probe at lnet_parse+0: -2 [17737.442026] Could not insert probe at lnet_send+0: -2 [17737.450235] Could not insert probe at lnet_send+0: -2 [17747.704105] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17747.713706] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17757.941575] Could not insert probe at lnet_parse+0: -2 [17757.949925] Could not insert probe at lnet_parse+0: -2 [17768.192842] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17768.202320] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17778.445888] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17778.456270] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17788.693263] Could not insert probe at lnet_send+0: -2 [17788.701164] Could not insert probe at lnet_send+0: -2 [17798.942950] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17798.952550] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17809.189821] Could not insert probe at lnet_parse+0: -2 [17809.197452] Could not insert probe at lnet_parse+0: -2 [17819.439828] Could not insert probe at lnet_send+0: -2 [17819.447362] Could not insert probe at lnet_send+0: -2 [17829.693791] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17829.702854] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17839.948140] Could not insert probe at lnet_post_send_locked+0: -2 [17839.955817] Could not insert probe at lnet_post_send_locked+0: -2 [17850.196938] Could not insert probe at lnet_send+0: -2 [17850.207004] Could not insert probe at lnet_send+0: -2 [17860.447304] Could not insert probe at lnet_parse+0: -2 [17860.457614] Could not insert probe at lnet_parse+0: -2 [17870.693323] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17870.701440] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17880.950898] Could not insert probe at lnet_parse+0: -2 [17880.959887] Could not insert probe at lnet_parse+0: -2 [17891.193181] Could not insert probe at lnet_send+0: -2 [17891.201067] Could not insert probe at lnet_send+0: -2 [17901.448584] Could not insert probe at lnet_parse+0: -2 [17901.456661] Could not insert probe at lnet_parse+0: -2 [17911.704813] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17911.713918] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [17921.949234] Could not insert probe at lnet_parse+0: -2 [17921.958492] Could not insert probe at lnet_parse+0: -2 [17932.205103] Could not insert probe at lnet_parse+0: -2 [17932.215881] Could not insert probe at lnet_parse+0: -2 [17942.464849] Could not insert probe at lnet_post_send_locked+0: -2 [17942.473913] Could not insert probe at lnet_post_send_locked+0: -2 [17952.697066] Could not insert probe at lnet_parse+0: -2 [17952.704689] Could not insert probe at lnet_parse+0: -2 [17962.953372] Could not insert probe at lnet_send+0: -2 [17962.963602] Could not insert probe at lnet_send+0: -2 [17973.217952] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17973.226319] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [17983.461148] Could not insert probe at lnet_send+0: -2 [17983.470715] Could not insert probe at lnet_send+0: -2 [17993.700057] Could not insert probe at lnet_send+0: -2 [17993.708277] Could not insert probe at lnet_send+0: -2 [18003.960394] Could not insert probe at lnet_post_send_locked+0: -2 [18003.969402] Could not insert probe at lnet_post_send_locked+0: -2 [18014.197943] Could not insert probe at lnet_parse+0: -2 [18014.207032] Could not insert probe at lnet_parse+0: -2 [18024.452999] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18024.461511] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18034.702296] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18034.711134] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18044.955638] Could not insert probe at lnet_send+0: -2 [18044.965794] Could not insert probe at lnet_send+0: -2 [18055.204417] Could not insert probe at lnet_parse+0: -2 [18055.212238] Could not insert probe at lnet_parse+0: -2 [18065.459130] Could not insert probe at lnet_parse+0: -2 [18065.467066] Could not insert probe at lnet_parse+0: -2 [18075.702086] Could not insert probe at lnet_send+0: -2 [18075.710042] Could not insert probe at lnet_send+0: -2 [18085.961362] Could not insert probe at lnet_post_send_locked+0: -2 [18085.970318] Could not insert probe at lnet_post_send_locked+0: -2 [18096.204528] Could not insert probe at lnet_parse+0: -2 [18096.212225] Could not insert probe at lnet_parse+0: -2 [18106.457706] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18106.465668] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18116.711064] Could not insert probe at lnet_parse+0: -2 [18116.719092] Could not insert probe at lnet_parse+0: -2 [18126.960884] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18126.970526] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18137.208969] Could not insert probe at lnet_post_send_locked+0: -2 [18137.217900] Could not insert probe at lnet_post_send_locked+0: -2 [18147.455701] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18147.464335] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18157.705496] Could not insert probe at lnet_post_send_locked+0: -2 [18157.714819] Could not insert probe at lnet_post_send_locked+0: -2 [18167.961540] Could not insert probe at lnet_parse+0: -2 [18167.969049] Could not insert probe at lnet_parse+0: -2 [18178.206267] Could not insert probe at lnet_parse+0: -2 [18178.215513] Could not insert probe at lnet_parse+0: -2 [18188.455783] Could not insert probe at lnet_parse+0: -2 [18188.463503] Could not insert probe at lnet_parse+0: -2 [18198.706352] Could not insert probe at lnet_send+0: -2 [18198.714356] Could not insert probe at lnet_send+0: -2 [18208.964340] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18208.973763] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18219.228567] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18219.238188] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18229.460538] Could not insert probe at lnet_send+0: -2 [18229.468285] Could not insert probe at lnet_send+0: -2 [18239.712267] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18239.720004] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18249.963436] Could not insert probe at lnet_send+0: -2 [18249.972478] Could not insert probe at lnet_send+0: -2 [18260.206507] Could not insert probe at lnet_parse+0: -2 [18260.214767] Could not insert probe at lnet_parse+0: -2 [18270.466622] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18270.475694] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18280.712503] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18280.720627] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18290.970001] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18290.980893] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18301.208773] Could not insert probe at lnet_post_send_locked+0: -2 [18301.218489] Could not insert probe at lnet_post_send_locked+0: -2 [18311.465255] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18311.474854] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18321.715675] Could not insert probe at lnet_send+0: -2 [18321.723262] Could not insert probe at lnet_send+0: -2 [18331.968264] Could not insert probe at lnet_parse+0: -2 [18331.977690] Could not insert probe at lnet_parse+0: -2 [18342.214734] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18342.222494] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18352.462615] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18352.470407] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18362.721060] Could not insert probe at lnet_post_send_locked+0: -2 [18362.731432] Could not insert probe at lnet_post_send_locked+0: -2 [18372.971642] Could not insert probe at lnet_parse+0: -2 [18372.979269] Could not insert probe at lnet_parse+0: -2 [18383.217665] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18383.226734] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18393.464841] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18393.472822] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18403.714405] Could not insert probe at lnet_parse+0: -2 [18403.722637] Could not insert probe at lnet_parse+0: -2 [18413.972241] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18413.981903] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18424.215906] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18424.223941] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18434.471558] Could not insert probe at lnet_parse+0: -2 [18434.479663] Could not insert probe at lnet_parse+0: -2 [18444.716976] Could not insert probe at lnet_parse+0: -2 [18444.724719] Could not insert probe at lnet_parse+0: -2 [18454.977136] Could not insert probe at lnet_post_send_locked+0: -2 [18454.985605] Could not insert probe at lnet_post_send_locked+0: -2 [18465.220529] Could not insert probe at lnet_parse+0: -2 [18465.229380] Could not insert probe at lnet_parse+0: -2 [18475.469154] Could not insert probe at lnet_parse+0: -2 [18475.477313] Could not insert probe at lnet_parse+0: -2 [18485.717225] Could not insert probe at lnet_parse+0: -2 [18485.724738] Could not insert probe at lnet_parse+0: -2 [18495.973499] Could not insert probe at lnet_parse+0: -2 [18495.981021] Could not insert probe at lnet_parse+0: -2 [18506.218210] Could not insert probe at lnet_parse+0: -2 [18506.227547] Could not insert probe at lnet_parse+0: -2 [18516.472973] Could not insert probe at lnet_parse+0: -2 [18516.481000] Could not insert probe at lnet_parse+0: -2 [18526.728272] Could not insert probe at lnet_send+0: -2 [18526.736389] Could not insert probe at lnet_send+0: -2 [18536.978889] Could not insert probe at lnet_post_send_locked+0: -2 [18536.989140] Could not insert probe at lnet_post_send_locked+0: -2 [18547.221195] Could not insert probe at lnet_parse+0: -2 [18547.228702] Could not insert probe at lnet_parse+0: -2 [18557.472126] Could not insert probe at lnet_parse+0: -2 [18557.480810] Could not insert probe at lnet_parse+0: -2 [18567.726355] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18567.735219] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18577.980202] Could not insert probe at lnet_post_send_locked+0: -2 [18577.988281] Could not insert probe at lnet_post_send_locked+0: -2 [18588.222611] Could not insert probe at lnet_parse+0: -2 [18588.231646] Could not insert probe at lnet_parse+0: -2 [18598.472813] Could not insert probe at lnet_parse+0: -2 [18598.480580] Could not insert probe at lnet_parse+0: -2 [18608.724775] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18608.732608] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18618.983098] Could not insert probe at lnet_parse+0: -2 [18618.992208] Could not insert probe at lnet_parse+0: -2 [18629.223526] Could not insert probe at lnet_post_send_locked+0: -2 [18629.233265] Could not insert probe at lnet_post_send_locked+0: -2 [18639.478350] Could not insert probe at lnet_parse+0: -2 [18639.485729] Could not insert probe at lnet_parse+0: -2 [18649.737094] Could not insert probe at lnet_parse+0: -2 [18649.745982] Could not insert probe at lnet_parse+0: -2 [18659.983121] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18659.991243] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18670.229190] Could not insert probe at lnet_parse+0: -2 [18670.239455] Could not insert probe at lnet_parse+0: -2 [18680.479275] Could not insert probe at lnet_parse+0: -2 [18680.487054] Could not insert probe at lnet_parse+0: -2 [18690.729862] Could not insert probe at lnet_parse+0: -2 [18690.737828] Could not insert probe at lnet_parse+0: -2 [18700.983783] Could not insert probe at lnet_parse+0: -2 [18700.991215] Could not insert probe at lnet_parse+0: -2 [18711.228900] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18711.236638] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18721.480623] Could not insert probe at lnet_parse+0: -2 [18721.489516] Could not insert probe at lnet_parse+0: -2 [18731.729422] Could not insert probe at lnet_parse+0: -2 [18731.737213] Could not insert probe at lnet_parse+0: -2 [18741.984954] Could not insert probe at lnet_parse+0: -2 [18741.992462] Could not insert probe at lnet_parse+0: -2 [18752.232675] Could not insert probe at lnet_parse+0: -2 [18752.242306] Could not insert probe at lnet_parse+0: -2 [18762.481029] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18762.489648] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18772.730841] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18772.739096] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18782.989342] Could not insert probe at lnet_parse+0: -2 [18782.998439] Could not insert probe at lnet_parse+0: -2 [18793.229550] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18793.237326] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [18803.483233] Could not insert probe at lnet_parse+0: -2 [18803.491266] Could not insert probe at lnet_parse+0: -2 [18813.734102] Could not insert probe at lnet_send+0: -2 [18813.741677] Could not insert probe at lnet_send+0: -2 [18823.988227] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18823.998201] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18834.232541] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18834.240360] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [18844.482313] Could not insert probe at lnet_parse+0: -2 [18844.489806] Could not insert probe at lnet_parse+0: -2 [18854.734111] Could not insert probe at lnet_parse+0: -2 [18854.743309] Could not insert probe at lnet_parse+0: -2 [18864.989524] Could not insert probe at lnet_parse+0: -2 [18864.997642] Could not insert probe at lnet_parse+0: -2 [18875.240590] Could not insert probe at lnet_parse+0: -2 [18875.250125] Could not insert probe at lnet_parse+0: -2 [18885.489766] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18885.497461] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18895.743159] Could not insert probe at lnet_parse+0: -2 [18895.753825] Could not insert probe at lnet_parse+0: -2 [18905.992808] Could not insert probe at lnet_send+0: -2 [18906.002382] Could not insert probe at lnet_send+0: -2 [18916.234645] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18916.242565] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [18926.492454] Could not insert probe at lnet_parse+0: -2 [18926.499988] Could not insert probe at lnet_parse+0: -2 [18936.737809] Could not insert probe at lnet_parse+0: -2 [18936.745454] Could not insert probe at lnet_parse+0: -2 [18946.995458] Could not insert probe at lnet_parse+0: -2 [18947.005001] Could not insert probe at lnet_parse+0: -2 [18957.238393] Could not insert probe at lnet_parse+0: -2 [18957.247230] Could not insert probe at lnet_parse+0: -2 [18967.494890] Could not insert probe at lnet_parse+0: -2 [18967.502732] Could not insert probe at lnet_parse+0: -2 [18977.744049] Could not insert probe at lnet_parse+0: -2 [18977.753149] Could not insert probe at lnet_parse+0: -2 [18987.995041] Could not insert probe at lnet_send+0: -2 [18988.002517] Could not insert probe at lnet_send+0: -2 [18998.237583] Could not insert probe at lnet_parse+0: -2 [18998.246813] Could not insert probe at lnet_parse+0: -2 [19008.494396] Could not insert probe at lnet_send+0: -2 [19008.502275] Could not insert probe at lnet_send+0: -2 [19018.744628] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19018.752764] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19028.996436] Could not insert probe at lnet_parse+0: -2 [19029.006798] Could not insert probe at lnet_parse+0: -2 [19039.243039] Could not insert probe at lnet_parse+0: -2 [19039.250453] Could not insert probe at lnet_parse+0: -2 [19049.490188] Could not insert probe at lnet_post_send_locked+0: -2 [19049.497872] Could not insert probe at lnet_post_send_locked+0: -2 [19059.742018] Could not insert probe at lnet_parse+0: -2 [19059.750399] Could not insert probe at lnet_parse+0: -2 [19069.999440] Could not insert probe at lnet_parse+0: -2 [19070.007838] Could not insert probe at lnet_parse+0: -2 [19080.247584] Could not insert probe at lnet_post_send_locked+0: -2 [19080.257176] Could not insert probe at lnet_post_send_locked+0: -2 [19090.495573] Could not insert probe at lnet_post_send_locked+0: -2 [19090.503577] Could not insert probe at lnet_post_send_locked+0: -2 [19100.744501] Could not insert probe at lnet_parse+0: -2 [19100.753221] Could not insert probe at lnet_parse+0: -2 [19110.999077] Could not insert probe at lnet_parse+0: -2 [19111.008412] Could not insert probe at lnet_parse+0: -2 [19121.242933] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19121.250698] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19131.498172] Could not insert probe at lnet_parse+0: -2 [19131.506147] Could not insert probe at lnet_parse+0: -2 [19141.753859] Could not insert probe at lnet_parse+0: -2 [19141.761525] Could not insert probe at lnet_parse+0: -2 [19152.003515] Could not insert probe at lnet_post_send_locked+0: -2 [19152.012466] Could not insert probe at lnet_post_send_locked+0: -2 [19162.249094] Could not insert probe at lnet_post_send_locked+0: -2 [19162.257387] Could not insert probe at lnet_post_send_locked+0: -2 [19172.499683] Could not insert probe at lnet_parse+0: -2 [19172.507735] Could not insert probe at lnet_parse+0: -2 [19182.750497] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19182.759150] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19193.005234] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19193.013649] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19203.250871] Could not insert probe at lnet_send+0: -2 [19203.258601] Could not insert probe at lnet_send+0: -2 [19213.504048] Could not insert probe at lnet_parse+0: -2 [19213.512237] Could not insert probe at lnet_parse+0: -2 [19223.746610] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19223.754526] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19234.004330] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19234.014229] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19244.248799] Could not insert probe at lnet_post_send_locked+0: -2 [19244.258400] Could not insert probe at lnet_post_send_locked+0: -2 [19254.500437] Could not insert probe at lnet_send+0: -2 [19254.507795] Could not insert probe at lnet_send+0: -2 [19264.755457] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19264.765369] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19275.010945] Could not insert probe at lnet_parse+0: -2 [19275.020230] Could not insert probe at lnet_parse+0: -2 [19285.253490] Could not insert probe at lnet_send+0: -2 [19285.261025] Could not insert probe at lnet_send+0: -2 [19295.505105] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19295.514258] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19305.753467] Could not insert probe at lnet_parse+0: -2 [19305.762788] Could not insert probe at lnet_parse+0: -2 [19316.003575] Could not insert probe at lnet_parse+0: -2 [19316.011329] Could not insert probe at lnet_parse+0: -2 [19326.253596] Could not insert probe at lnet_post_send_locked+0: -2 [19326.262675] Could not insert probe at lnet_post_send_locked+0: -2 [19336.510772] Could not insert probe at lnet_post_send_locked+0: -2 [19336.520028] Could not insert probe at lnet_post_send_locked+0: -2 [19346.761414] Could not insert probe at lnet_parse+0: -2 [19346.770766] Could not insert probe at lnet_parse+0: -2 [19357.009863] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19357.019031] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19367.255544] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19367.265336] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19377.510620] Could not insert probe at lnet_post_send_locked+0: -2 [19377.519765] Could not insert probe at lnet_post_send_locked+0: -2 [19387.765634] Could not insert probe at lnet_post_send_locked+0: -2 [19387.775274] Could not insert probe at lnet_post_send_locked+0: -2 [19398.013028] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19398.022608] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19408.267685] Could not insert probe at lnet_send+0: -2 [19408.276788] Could not insert probe at lnet_send+0: -2 [19418.504064] Could not insert probe at lnet_parse+0: -2 [19418.513391] Could not insert probe at lnet_parse+0: -2 [19428.773446] Could not insert probe at lnet_parse+0: -2 [19428.782555] Could not insert probe at lnet_parse+0: -2 [19439.008658] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19439.018283] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19449.268129] Could not insert probe at lnet_post_send_locked+0: -2 [19449.277507] Could not insert probe at lnet_post_send_locked+0: -2 [19459.515744] Could not insert probe at lnet_parse+0: -2 [19459.524681] Could not insert probe at lnet_parse+0: -2 [19469.773404] Could not insert probe at lnet_parse+0: -2 [19469.782452] Could not insert probe at lnet_parse+0: -2 [19480.015915] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19480.025780] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19490.271718] Could not insert probe at lnet_parse+0: -2 [19490.281202] Could not insert probe at lnet_parse+0: -2 [19500.516626] Could not insert probe at lnet_parse+0: -2 [19500.525571] Could not insert probe at lnet_parse+0: -2 [19510.768811] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19510.779074] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19521.014483] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19521.022417] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19531.264353] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19531.274125] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19541.516684] Could not insert probe at lnet_send+0: -2 [19541.526226] Could not insert probe at lnet_send+0: -2 [19551.770155] Could not insert probe at lnet_parse+0: -2 [19551.779581] Could not insert probe at lnet_parse+0: -2 [19562.021728] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19562.031160] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19572.268560] Could not insert probe at lnet_post_send_locked+0: -2 [19572.278259] Could not insert probe at lnet_post_send_locked+0: -2 [19582.513342] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19582.522884] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19592.770271] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19592.780294] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19603.024576] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19603.035655] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19613.270729] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19613.280345] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19623.527432] Could not insert probe at lnet_parse+0: -2 [19623.536246] Could not insert probe at lnet_parse+0: -2 [19633.765859] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19633.776375] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19644.019392] Could not insert probe at lnet_parse+0: -2 [19644.027346] Could not insert probe at lnet_parse+0: -2 [19654.267965] Could not insert probe at lnet_parse+0: -2 [19654.277523] Could not insert probe at lnet_parse+0: -2 [19664.513072] Could not insert probe at lnet_parse+0: -2 [19664.520846] Could not insert probe at lnet_parse+0: -2 [19674.770390] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19674.778372] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19685.020463] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19685.028794] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19695.270173] Could not insert probe at lnet_parse+0: -2 [19695.278189] Could not insert probe at lnet_parse+0: -2 [19705.527430] Could not insert probe at lnet_parse+0: -2 [19705.535585] Could not insert probe at lnet_parse+0: -2 [19715.765928] Could not insert probe at lnet_parse+0: -2 [19715.773835] Could not insert probe at lnet_parse+0: -2 [19726.020130] Could not insert probe at lnet_parse+0: -2 [19726.029400] Could not insert probe at lnet_parse+0: -2 [19736.275331] Could not insert probe at lnet_parse+0: -2 [19736.284031] Could not insert probe at lnet_parse+0: -2 [19746.520957] Could not insert probe at lnet_send+0: -2 [19746.529271] Could not insert probe at lnet_send+0: -2 [19756.771478] Could not insert probe at lnet_send+0: -2 [19756.779513] Could not insert probe at lnet_send+0: -2 [19767.025979] Could not insert probe at lnet_parse+0: -2 [19767.034134] Could not insert probe at lnet_parse+0: -2 [19777.275323] Could not insert probe at lnet_post_send_locked+0: -2 [19777.284381] Could not insert probe at lnet_post_send_locked+0: -2 [19787.525187] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19787.533205] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19797.772480] Could not insert probe at lnet_parse+0: -2 [19797.780052] Could not insert probe at lnet_parse+0: -2 [19808.030063] Could not insert probe at lnet_parse+0: -2 [19808.037620] Could not insert probe at lnet_parse+0: -2 [19818.280694] Could not insert probe at lnet_parse+0: -2 [19818.290113] Could not insert probe at lnet_parse+0: -2 [19828.527877] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19828.537375] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19838.781140] Could not insert probe at lnet_parse+0: -2 [19838.790769] Could not insert probe at lnet_parse+0: -2 [19849.030667] Could not insert probe at lnet_parse+0: -2 [19849.040347] Could not insert probe at lnet_parse+0: -2 [19859.285073] Could not insert probe at lnet_parse+0: -2 [19859.294570] Could not insert probe at lnet_parse+0: -2 [19869.530936] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19869.540948] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19879.777554] Could not insert probe at lnet_send+0: -2 [19879.785490] Could not insert probe at lnet_send+0: -2 [19890.022261] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19890.031793] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19900.287267] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19900.297107] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19910.532337] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19910.541491] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19920.788568] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19920.798066] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19931.033963] Could not insert probe at lnet_send+0: -2 [19931.043444] Could not insert probe at lnet_send+0: -2 [19941.279974] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19941.289694] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [19951.530647] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19951.540260] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [19961.785261] Could not insert probe at lnet_parse+0: -2 [19961.794500] Could not insert probe at lnet_parse+0: -2 [19972.036346] Could not insert probe at lnet_parse+0: -2 [19972.046083] Could not insert probe at lnet_parse+0: -2 [19982.284120] Could not insert probe at lnet_parse+0: -2 [19982.293310] Could not insert probe at lnet_parse+0: -2 [19992.530412] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [19992.538830] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20002.786144] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20002.797106] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20013.027907] Could not insert probe at lnet_send+0: -2 [20013.035582] Could not insert probe at lnet_send+0: -2 [20023.286052] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20023.295862] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20033.530103] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20033.538202] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20043.787755] Could not insert probe at lnet_parse+0: -2 [20043.796745] Could not insert probe at lnet_parse+0: -2 [20054.046671] Could not insert probe at lnet_parse+0: -2 [20054.056620] Could not insert probe at lnet_parse+0: -2 [20064.288668] Could not insert probe at lnet_post_send_locked+0: -2 [20064.297694] Could not insert probe at lnet_post_send_locked+0: -2 [20074.544057] Could not insert probe at lnet_parse+0: -2 [20074.553106] Could not insert probe at lnet_parse+0: -2 [20084.788451] Could not insert probe at lnet_parse+0: -2 [20084.799232] Could not insert probe at lnet_parse+0: -2 [20095.047925] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20095.057760] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20105.285210] Could not insert probe at lnet_parse+0: -2 [20105.294038] Could not insert probe at lnet_parse+0: -2 [20115.536714] Could not insert probe at lnet_post_send_locked+0: -2 [20115.546559] Could not insert probe at lnet_post_send_locked+0: -2 [20125.791714] Could not insert probe at lnet_parse+0: -2 [20125.801246] Could not insert probe at lnet_parse+0: -2 [20136.046017] Could not insert probe at lnet_parse+0: -2 [20136.055147] Could not insert probe at lnet_parse+0: -2 [20146.291607] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20146.300668] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20156.546485] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20156.556062] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20166.794250] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20166.802346] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20177.045393] Could not insert probe at lnet_send+0: -2 [20177.055003] Could not insert probe at lnet_send+0: -2 [20187.294526] Could not insert probe at lnet_parse+0: -2 [20187.305264] Could not insert probe at lnet_parse+0: -2 [20197.543037] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20197.553309] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20207.792856] Could not insert probe at lnet_parse+0: -2 [20207.802098] Could not insert probe at lnet_parse+0: -2 [20218.034675] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20218.044758] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20228.297762] Could not insert probe at lnet_send+0: -2 [20228.306608] Could not insert probe at lnet_send+0: -2 [20238.537820] Could not insert probe at lnet_parse+0: -2 [20238.547324] Could not insert probe at lnet_parse+0: -2 [20248.793786] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20248.804752] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20259.046527] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20259.055987] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20269.301980] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20269.311739] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20279.547437] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20279.556830] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20289.800135] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20289.809248] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20300.047814] Could not insert probe at lnet_parse+0: -2 [20300.058621] Could not insert probe at lnet_parse+0: -2 [20310.299468] Could not insert probe at lnet_parse+0: -2 [20310.308864] Could not insert probe at lnet_parse+0: -2 [20320.548889] Could not insert probe at lnet_post_send_locked+0: -2 [20320.558633] Could not insert probe at lnet_post_send_locked+0: -2 [20330.805471] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20330.814810] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20341.049871] Could not insert probe at lnet_send+0: -2 [20341.059176] Could not insert probe at lnet_send+0: -2 [20351.298381] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20351.307779] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20361.548613] Could not insert probe at lnet_parse+0: -2 [20361.557742] Could not insert probe at lnet_parse+0: -2 [20371.799738] Could not insert probe at lnet_parse+0: -2 [20371.810002] Could not insert probe at lnet_parse+0: -2 [20382.042568] Could not insert probe at lnet_post_send_locked+0: -2 [20382.051841] Could not insert probe at lnet_post_send_locked+0: -2 [20392.304057] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20392.315281] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20402.552038] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20402.561672] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20412.803354] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20412.811051] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20423.048573] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20423.056359] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20433.301193] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20433.310967] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20443.550430] Could not insert probe at lnet_parse+0: -2 [20443.558224] Could not insert probe at lnet_parse+0: -2 [20453.798985] Could not insert probe at lnet_parse+0: -2 [20453.806607] Could not insert probe at lnet_parse+0: -2 [20464.054882] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20464.064063] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20474.298733] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20474.306438] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20484.556322] Could not insert probe at lnet_parse+0: -2 [20484.563917] Could not insert probe at lnet_parse+0: -2 [20494.799781] Could not insert probe at lnet_parse+0: -2 [20494.809331] Could not insert probe at lnet_parse+0: -2 [20505.053231] Could not insert probe at lnet_send+0: -2 [20505.061180] Could not insert probe at lnet_send+0: -2 [20515.304285] Could not insert probe at lnet_send+0: -2 [20515.314169] Could not insert probe at lnet_send+0: -2 [20525.553289] Could not insert probe at lnet_parse+0: -2 [20525.561426] Could not insert probe at lnet_parse+0: -2 [20535.798776] Could not insert probe at lnet_send+0: -2 [20535.806736] Could not insert probe at lnet_send+0: -2 [20546.055394] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20546.063273] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20556.302883] Could not insert probe at lnet_parse+0: -2 [20556.312834] Could not insert probe at lnet_parse+0: -2 [20566.558134] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20566.566096] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20576.808004] Could not insert probe at lnet_parse+0: -2 [20576.817527] Could not insert probe at lnet_parse+0: -2 [20587.056229] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20587.065217] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20597.307670] Could not insert probe at lnet_parse+0: -2 [20597.317293] Could not insert probe at lnet_parse+0: -2 [20607.554470] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20607.562632] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20617.806179] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20617.814034] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20628.055300] Could not insert probe at lnet_parse+0: -2 [20628.063378] Could not insert probe at lnet_parse+0: -2 [20638.307979] Could not insert probe at lnet_send+0: -2 [20638.315777] Could not insert probe at lnet_send+0: -2 [20648.557913] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20648.566291] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20658.803626] Could not insert probe at lnet_send+0: -2 [20658.811336] Could not insert probe at lnet_send+0: -2 [20669.063115] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20669.072731] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20679.308376] Could not insert probe at lnet_parse+0: -2 [20679.317391] Could not insert probe at lnet_parse+0: -2 [20689.552620] Could not insert probe at lnet_parse+0: -2 [20689.561754] Could not insert probe at lnet_parse+0: -2 [20699.806049] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20699.813991] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20710.058208] Could not insert probe at lnet_send+0: -2 [20710.067626] Could not insert probe at lnet_send+0: -2 [20720.311872] Could not insert probe at lnet_post_send_locked+0: -2 [20720.319631] Could not insert probe at lnet_post_send_locked+0: -2 [20730.559703] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20730.567345] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20740.808649] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20740.816619] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20751.056302] Could not insert probe at lnet_send+0: -2 [20751.064198] Could not insert probe at lnet_send+0: -2 [20761.312262] Could not insert probe at lnet_parse+0: -2 [20761.321642] Could not insert probe at lnet_parse+0: -2 [20771.558303] Could not insert probe at lnet_post_send_locked+0: -2 [20771.566113] Could not insert probe at lnet_post_send_locked+0: -2 [20781.815059] Could not insert probe at lnet_post_send_locked+0: -2 [20781.823391] Could not insert probe at lnet_post_send_locked+0: -2 [20792.063033] Could not insert probe at lnet_parse+0: -2 [20792.070832] Could not insert probe at lnet_parse+0: -2 [20802.316471] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20802.326730] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20812.562387] Could not insert probe at lnet_send+0: -2 [20812.570686] Could not insert probe at lnet_send+0: -2 [20822.810697] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20822.818732] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20833.064691] Could not insert probe at lnet_parse+0: -2 [20833.074835] Could not insert probe at lnet_parse+0: -2 [20843.315177] Could not insert probe at lnet_send+0: -2 [20843.322843] Could not insert probe at lnet_send+0: -2 [20853.564423] Could not insert probe at lnet_send+0: -2 [20853.572194] Could not insert probe at lnet_send+0: -2 [20863.813022] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20863.822694] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20874.071035] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20874.081845] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [20884.319140] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20884.327590] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20894.562401] Could not insert probe at lnet_send+0: -2 [20894.569958] Could not insert probe at lnet_send+0: -2 [20904.813385] Could not insert probe at lnet_parse+0: -2 [20904.821245] Could not insert probe at lnet_parse+0: -2 [20915.068858] Could not insert probe at lnet_parse+0: -2 [20915.079124] Could not insert probe at lnet_parse+0: -2 [20925.318450] Could not insert probe at lnet_send+0: -2 [20925.328012] Could not insert probe at lnet_send+0: -2 [20935.566885] Could not insert probe at lnet_parse+0: -2 [20935.574488] Could not insert probe at lnet_parse+0: -2 [20945.813805] Could not insert probe at lnet_parse+0: -2 [20945.822886] Could not insert probe at lnet_parse+0: -2 [20956.069776] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20956.078363] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [20966.319457] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20966.328638] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [20976.569595] Could not insert probe at lnet_post_send_locked+0: -2 [20976.577581] Could not insert probe at lnet_post_send_locked+0: -2 [20986.826116] Could not insert probe at lnet_send+0: -2 [20986.834506] Could not insert probe at lnet_send+0: -2 [20997.069306] Could not insert probe at lnet_post_send_locked+0: -2 [20997.078921] Could not insert probe at lnet_post_send_locked+0: -2 [21007.326011] Could not insert probe at lnet_post_send_locked+0: -2 [21007.335049] Could not insert probe at lnet_post_send_locked+0: -2 [21017.571648] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21017.579746] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21027.818656] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21027.827432] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21038.076571] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21038.086194] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21048.322569] Could not insert probe at lnet_send+0: -2 [21048.330322] Could not insert probe at lnet_send+0: -2 [21058.570407] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21058.578156] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21068.824038] Could not insert probe at lnet_parse+0: -2 [21068.831756] Could not insert probe at lnet_parse+0: -2 [21079.068519] Could not insert probe at lnet_parse+0: -2 [21079.076050] Could not insert probe at lnet_parse+0: -2 [21089.329711] Could not insert probe at lnet_parse+0: -2 [21089.337667] Could not insert probe at lnet_parse+0: -2 [21099.579426] Could not insert probe at lnet_parse+0: -2 [21099.588753] Could not insert probe at lnet_parse+0: -2 [21109.821658] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21109.830472] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21120.074253] Could not insert probe at lnet_parse+0: -2 [21120.083752] Could not insert probe at lnet_parse+0: -2 [21130.325520] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [21130.333656] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [21140.573110] Could not insert probe at lnet_send+0: -2 [21140.581748] Could not insert probe at lnet_send+0: -2 [21150.829009] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [21150.839404] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [21161.072840] Could not insert probe at lnet_parse+0: -2 [21161.080350] Could not insert probe at lnet_parse+0: -2 [21171.327559] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21171.336725] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21181.573800] Could not insert probe at lnet_parse+0: -2 [21181.581625] Could not insert probe at lnet_parse+0: -2 [21191.830592] Could not insert probe at lnet_post_send_locked+0: -2 [21191.838550] Could not insert probe at lnet_post_send_locked+0: -2 [21202.080318] Could not insert probe at lnet_parse+0: -2 [21202.090439] Could not insert probe at lnet_parse+0: -2 [21212.331131] Could not insert probe at lnet_post_send_locked+0: -2 [21212.339321] Could not insert probe at lnet_post_send_locked+0: -2 [21222.579433] Could not insert probe at lnet_post_send_locked+0: -2 [21222.587700] Could not insert probe at lnet_post_send_locked+0: -2 [21232.826918] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21232.835134] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21243.078725] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21243.086593] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21253.330510] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [21253.339995] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [21263.576689] Could not insert probe at lnet_post_send_locked+0: -2 [21263.584344] Could not insert probe at lnet_post_send_locked+0: -2 [21273.833058] Could not insert probe at lnet_post_send_locked+0: -2 [21273.842136] Could not insert probe at lnet_post_send_locked+0: -2 [21284.081667] Could not insert probe at lnet_parse+0: -2 [21284.089206] Could not insert probe at lnet_parse+0: -2 [21294.333266] Could not insert probe at lnet_parse+0: -2 [21294.342605] Could not insert probe at lnet_parse+0: -2 [21304.576681] Could not insert probe at lnet_send+0: -2 [21304.584925] Could not insert probe at lnet_send+0: -2 [21314.837780] Could not insert probe at lnet_parse+0: -2 [21314.847455] Could not insert probe at lnet_parse+0: -2 [21325.094611] Could not insert probe at lnet_parse+0: -2 [21325.102914] Could not insert probe at lnet_parse+0: -2 [21335.339319] Could not insert probe at lnet_post_send_locked+0: -2 [21335.349261] Could not insert probe at lnet_post_send_locked+0: -2 [21345.594308] Could not insert probe at lnet_post_send_locked+0: -2 [21345.603504] Could not insert probe at lnet_post_send_locked+0: -2 [21355.839134] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [21355.849159] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [21366.092527] Could not insert probe at lnet_parse+0: -2 [21366.103477] Could not insert probe at lnet_parse+0: -2 [21376.340369] Could not insert probe at lnet_send+0: -2 [21376.349984] Could not insert probe at lnet_send+0: -2 [21386.594475] Could not insert probe at lnet_parse+0: -2 [21386.604317] Could not insert probe at lnet_parse+0: -2 [21396.839383] Could not insert probe at lnet_send+0: -2 [21396.848680] Could not insert probe at lnet_send+0: -2 [21407.096283] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21407.105820] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21417.345122] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21417.354503] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21427.596553] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21427.605732] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21437.842527] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21437.852165] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21448.092706] Could not insert probe at lnet_parse+0: -2 [21448.104093] Could not insert probe at lnet_parse+0: -2 [21458.345396] Could not insert probe at lnet_send+0: -2 [21458.355031] Could not insert probe at lnet_send+0: -2 [21468.594107] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21468.603285] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21478.843848] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [21478.853815] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [21489.087942] Could not insert probe at lnet_parse+0: -2 [21489.096298] Could not insert probe at lnet_parse+0: -2 [21499.343332] Could not insert probe at lnet_parse+0: -2 [21499.352745] Could not insert probe at lnet_parse+0: -2 [21509.595761] Could not insert probe at lnet_send+0: -2 [21509.605323] Could not insert probe at lnet_send+0: -2 [21519.850698] Could not insert probe at lnet_parse+0: -2 [21519.861436] Could not insert probe at lnet_parse+0: -2 [21530.093166] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [21530.102290] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [21540.352238] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21540.363258] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21550.590320] Could not insert probe at lnet_send+0: -2 [21550.598232] Could not insert probe at lnet_send+0: -2 [21560.856715] Could not insert probe at lnet_parse+0: -2 [21560.865986] Could not insert probe at lnet_parse+0: -2 [21571.098265] Could not insert probe at lnet_post_send_locked+0: -2 [21571.107567] Could not insert probe at lnet_post_send_locked+0: -2 [21581.356303] Could not insert probe at lnet_parse+0: -2 [21581.365051] Could not insert probe at lnet_parse+0: -2 [21591.599068] Could not insert probe at lnet_parse+0: -2 [21591.608434] Could not insert probe at lnet_parse+0: -2 [21601.854933] Could not insert probe at lnet_parse+0: -2 [21601.864767] Could not insert probe at lnet_parse+0: -2 [21612.098989] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21612.109102] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21622.355283] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21622.364724] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21632.592751] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21632.602974] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21642.853232] Could not insert probe at lnet_post_send_locked+0: -2 [21642.863493] Could not insert probe at lnet_post_send_locked+0: -2 [21653.102249] Could not insert probe at lnet_parse+0: -2 [21653.110678] Could not insert probe at lnet_parse+0: -2 [21663.349003] Could not insert probe at lnet_post_send_locked+0: -2 [21663.357087] Could not insert probe at lnet_post_send_locked+0: -2 [21673.600916] Could not insert probe at lnet_parse+0: -2 [21673.608266] Could not insert probe at lnet_parse+0: -2 [21683.849129] Could not insert probe at lnet_post_send_locked+0: -2 [21683.856932] Could not insert probe at lnet_post_send_locked+0: -2 [21694.103191] Could not insert probe at lnet_parse+0: -2 [21694.112407] Could not insert probe at lnet_parse+0: -2 [21704.347669] Could not insert probe at lnet_parse+0: -2 [21704.356630] Could not insert probe at lnet_parse+0: -2 [21714.597949] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21714.605788] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21724.842848] Could not insert probe at lnet_post_send_locked+0: -2 [21724.852530] Could not insert probe at lnet_post_send_locked+0: -2 [21735.105408] Could not insert probe at lnet_send+0: -2 [21735.115174] Could not insert probe at lnet_send+0: -2 [21745.354031] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [21745.363615] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [21755.606223] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21755.615832] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21765.852321] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21765.861955] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21776.107406] Could not insert probe at lnet_send+0: -2 [21776.116515] Could not insert probe at lnet_send+0: -2 [21786.356821] Could not insert probe at lnet_send+0: -2 [21786.365824] Could not insert probe at lnet_send+0: -2 [21796.605367] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21796.613385] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21806.856227] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21806.865764] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21817.112041] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21817.123142] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [21827.355608] Could not insert probe at lnet_parse+0: -2 [21827.363494] Could not insert probe at lnet_parse+0: -2 [21837.613332] Could not insert probe at lnet_parse+0: -2 [21837.620929] Could not insert probe at lnet_parse+0: -2 [21847.859167] Could not insert probe at lnet_parse+0: -2 [21847.868518] Could not insert probe at lnet_parse+0: -2 [21858.110144] Could not insert probe at lnet_parse+0: -2 [21858.119812] Could not insert probe at lnet_parse+0: -2 [21868.359617] Could not insert probe at lnet_parse+0: -2 [21868.369016] Could not insert probe at lnet_parse+0: -2 [21878.610861] Could not insert probe at lnet_post_send_locked+0: -2 [21878.620439] Could not insert probe at lnet_post_send_locked+0: -2 [21888.860383] Could not insert probe at lnet_parse+0: -2 [21888.869772] Could not insert probe at lnet_parse+0: -2 [21899.108766] Could not insert probe at lnet_send+0: -2 [21899.118152] Could not insert probe at lnet_send+0: -2 [21909.362217] Could not insert probe at lnet_post_send_locked+0: -2 [21909.371725] Could not insert probe at lnet_post_send_locked+0: -2 [21919.612216] Could not insert probe at lnet_parse+0: -2 [21919.620850] Could not insert probe at lnet_parse+0: -2 [21929.863828] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21929.873542] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21940.109259] Could not insert probe at lnet_parse+0: -2 [21940.116883] Could not insert probe at lnet_parse+0: -2 [21950.366386] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21950.376255] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [21960.612464] Could not insert probe at lnet_post_send_locked+0: -2 [21960.621381] Could not insert probe at lnet_post_send_locked+0: -2 [21970.864692] Could not insert probe at lnet_parse+0: -2 [21970.874206] Could not insert probe at lnet_parse+0: -2 [21981.113268] Could not insert probe at lnet_parse+0: -2 [21981.122551] Could not insert probe at lnet_parse+0: -2 [21991.369929] Could not insert probe at lnet_post_send_locked+0: -2 [21991.380864] Could not insert probe at lnet_post_send_locked+0: -2 [22001.613816] Could not insert probe at lnet_parse+0: -2 [22001.623195] Could not insert probe at lnet_parse+0: -2 [22011.861098] Could not insert probe at lnet_parse+0: -2 [22011.870732] Could not insert probe at lnet_parse+0: -2 [22022.110384] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22022.118061] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22032.369927] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22032.379857] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22042.616353] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22042.625847] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22052.864972] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22052.873221] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22063.116663] Could not insert probe at lnet_send+0: -2 [22063.125592] Could not insert probe at lnet_send+0: -2 [22073.369281] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22073.379032] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22083.618378] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22083.627196] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22093.860792] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22093.868751] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22104.117339] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22104.125310] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22114.365060] Could not insert probe at lnet_post_send_locked+0: -2 [22114.374713] Could not insert probe at lnet_post_send_locked+0: -2 [22124.615137] Could not insert probe at lnet_parse+0: -2 [22124.624100] Could not insert probe at lnet_parse+0: -2 [22134.861341] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22134.869529] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22145.117580] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22145.126912] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22155.364475] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22155.372217] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22165.627439] Could not insert probe at lnet_post_send_locked+0: -2 [22165.635682] Could not insert probe at lnet_post_send_locked+0: -2 [22175.869431] Could not insert probe at lnet_send+0: -2 [22175.879675] Could not insert probe at lnet_send+0: -2 [22186.119623] Could not insert probe at lnet_post_send_locked+0: -2 [22186.127512] Could not insert probe at lnet_post_send_locked+0: -2 [22196.367688] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22196.376769] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22206.614843] Could not insert probe at lnet_parse+0: -2 [22206.622354] Could not insert probe at lnet_parse+0: -2 [22216.872758] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22216.880715] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22227.120545] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22227.128572] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22237.364849] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22237.374364] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22247.615386] Could not insert probe at lnet_send+0: -2 [22247.622929] Could not insert probe at lnet_send+0: -2 [22257.870128] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22257.878311] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22268.128130] Could not insert probe at lnet_parse+0: -2 [22268.137550] Could not insert probe at lnet_parse+0: -2 [22278.365359] Could not insert probe at lnet_post_send_locked+0: -2 [22278.374967] Could not insert probe at lnet_post_send_locked+0: -2 [22288.616901] Could not insert probe at lnet_post_send_locked+0: -2 [22288.624517] Could not insert probe at lnet_post_send_locked+0: -2 [22298.874256] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22298.881797] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22309.125498] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22309.133437] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22319.368928] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22319.376733] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22329.620547] Could not insert probe at lnet_parse+0: -2 [22329.628172] Could not insert probe at lnet_parse+0: -2 [22339.870671] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22339.878421] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22350.123542] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22350.132958] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22360.370356] Could not insert probe at lnet_post_send_locked+0: -2 [22360.378221] Could not insert probe at lnet_post_send_locked+0: -2 [22370.621790] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22370.629615] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22380.872463] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22380.882232] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22391.126474] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22391.134602] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22401.375443] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22401.385030] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22411.622417] Could not insert probe at lnet_parse+0: -2 [22411.630387] Could not insert probe at lnet_parse+0: -2 [22421.883464] Could not insert probe at lnet_parse+0: -2 [22421.891868] Could not insert probe at lnet_parse+0: -2 [22432.123541] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22432.131441] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22442.373769] Could not insert probe at lnet_parse+0: -2 [22442.381604] Could not insert probe at lnet_parse+0: -2 [22452.625960] Could not insert probe at lnet_parse+0: -2 [22452.633993] Could not insert probe at lnet_parse+0: -2 [22462.878214] Could not insert probe at lnet_post_send_locked+0: -2 [22462.886406] Could not insert probe at lnet_post_send_locked+0: -2 [22473.128982] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22473.138626] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22483.374330] Could not insert probe at lnet_parse+0: -2 [22483.381806] Could not insert probe at lnet_parse+0: -2 [22493.628524] Could not insert probe at lnet_parse+0: -2 [22493.636603] Could not insert probe at lnet_parse+0: -2 [22503.881459] Could not insert probe at lnet_parse+0: -2 [22503.891529] Could not insert probe at lnet_parse+0: -2 [22514.133420] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22514.141431] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22524.380015] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22524.390304] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22534.626139] Could not insert probe at lnet_parse+0: -2 [22534.634169] Could not insert probe at lnet_parse+0: -2 [22544.882857] Could not insert probe at lnet_parse+0: -2 [22544.890455] Could not insert probe at lnet_parse+0: -2 [22555.134542] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22555.142578] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22565.379636] Could not insert probe at lnet_send+0: -2 [22565.389582] Could not insert probe at lnet_send+0: -2 [22575.626849] Could not insert probe at lnet_send+0: -2 [22575.634719] Could not insert probe at lnet_send+0: -2 [22585.889880] Could not insert probe at lnet_parse+0: -2 [22585.898767] Could not insert probe at lnet_parse+0: -2 [22596.129591] Could not insert probe at lnet_parse+0: -2 [22596.138519] Could not insert probe at lnet_parse+0: -2 [22606.378366] Could not insert probe at lnet_parse+0: -2 [22606.385976] Could not insert probe at lnet_parse+0: -2 [22616.634819] Could not insert probe at lnet_send+0: -2 [22616.642347] Could not insert probe at lnet_send+0: -2 [22626.888954] Could not insert probe at lnet_parse+0: -2 [22626.896833] Could not insert probe at lnet_parse+0: -2 [22637.138173] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22637.146239] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22647.379856] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22647.387522] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22657.645362] Could not insert probe at lnet_parse+0: -2 [22657.655082] Could not insert probe at lnet_parse+0: -2 [22667.893926] Could not insert probe at lnet_parse+0: -2 [22667.901558] Could not insert probe at lnet_parse+0: -2 [22678.133328] Could not insert probe at lnet_parse+0: -2 [22678.140750] Could not insert probe at lnet_parse+0: -2 [22688.393341] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22688.402135] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22698.637106] Could not insert probe at lnet_post_send_locked+0: -2 [22698.645028] Could not insert probe at lnet_post_send_locked+0: -2 [22708.895090] Could not insert probe at lnet_parse+0: -2 [22708.906172] Could not insert probe at lnet_parse+0: -2 [22719.135949] Could not insert probe at lnet_parse+0: -2 [22719.143440] Could not insert probe at lnet_parse+0: -2 [22729.396036] Could not insert probe at lnet_post_send_locked+0: -2 [22729.405884] Could not insert probe at lnet_post_send_locked+0: -2 [22739.643514] Could not insert probe at lnet_post_send_locked+0: -2 [22739.653388] Could not insert probe at lnet_post_send_locked+0: -2 [22749.893407] Could not insert probe at lnet_post_send_locked+0: -2 [22749.901673] Could not insert probe at lnet_post_send_locked+0: -2 [22760.141062] Could not insert probe at lnet_parse+0: -2 [22760.148849] Could not insert probe at lnet_parse+0: -2 [22770.397964] Could not insert probe at lnet_parse+0: -2 [22770.407428] Could not insert probe at lnet_parse+0: -2 [22780.645066] Could not insert probe at lnet_send+0: -2 [22780.655862] Could not insert probe at lnet_send+0: -2 [22790.894598] Could not insert probe at lnet_post_send_locked+0: -2 [22790.902431] Could not insert probe at lnet_post_send_locked+0: -2 [22801.145207] Could not insert probe at lnet_parse+0: -2 [22801.154811] Could not insert probe at lnet_parse+0: -2 [22811.402987] Could not insert probe at lnet_send+0: -2 [22811.412030] Could not insert probe at lnet_send+0: -2 [22821.639117] Could not insert probe at lnet_send+0: -2 [22821.646449] Could not insert probe at lnet_send+0: -2 [22831.898909] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22831.907063] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22842.149765] Could not insert probe at lnet_send+0: -2 [22842.157341] Could not insert probe at lnet_send+0: -2 [22852.397308] Could not insert probe at lnet_send+0: -2 [22852.405128] Could not insert probe at lnet_send+0: -2 [22862.643476] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22862.652076] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22872.894742] Could not insert probe at lnet_post_send_locked+0: -2 [22872.904950] Could not insert probe at lnet_post_send_locked+0: -2 [22883.149775] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22883.158826] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22893.392915] Could not insert probe at lnet_parse+0: -2 [22893.401184] Could not insert probe at lnet_parse+0: -2 [22903.642244] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22903.650557] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22913.899455] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22913.907160] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [22924.146366] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22924.154484] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [22934.396276] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22934.406064] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [22944.643572] Could not insert probe at lnet_parse+0: -2 [22944.651313] Could not insert probe at lnet_parse+0: -2 [22954.899665] Could not insert probe at lnet_send+0: -2 [22954.907407] Could not insert probe at lnet_send+0: -2 [22965.145215] Could not insert probe at lnet_parse+0: -2 [22965.152915] Could not insert probe at lnet_parse+0: -2 [22975.395311] Could not insert probe at lnet_parse+0: -2 [22975.404568] Could not insert probe at lnet_parse+0: -2 [22985.655262] Could not insert probe at lnet_parse+0: -2 [22985.662939] Could not insert probe at lnet_parse+0: -2 [22995.903162] Could not insert probe at lnet_send+0: -2 [22995.910975] Could not insert probe at lnet_send+0: -2 [23006.154948] Could not insert probe at lnet_parse+0: -2 [23006.164776] Could not insert probe at lnet_parse+0: -2 [23016.398254] Could not insert probe at lnet_parse+0: -2 [23016.406102] Could not insert probe at lnet_parse+0: -2 [23026.646479] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23026.654452] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23036.904280] Could not insert probe at lnet_parse+0: -2 [23036.914553] Could not insert probe at lnet_parse+0: -2 [23047.152255] Could not insert probe at lnet_send+0: -2 [23047.160355] Could not insert probe at lnet_send+0: -2 [23057.401174] Could not insert probe at lnet_parse+0: -2 [23057.410685] Could not insert probe at lnet_parse+0: -2 [23067.647244] Could not insert probe at lnet_send+0: -2 [23067.655183] Could not insert probe at lnet_send+0: -2 [23077.903711] Could not insert probe at lnet_parse+0: -2 [23077.911571] Could not insert probe at lnet_parse+0: -2 [23088.152077] Could not insert probe at lnet_post_send_locked+0: -2 [23088.159850] Could not insert probe at lnet_post_send_locked+0: -2 [23098.399281] Could not insert probe at lnet_parse+0: -2 [23098.408260] Could not insert probe at lnet_parse+0: -2 [23108.647588] Could not insert probe at lnet_parse+0: -2 [23108.655330] Could not insert probe at lnet_parse+0: -2 [23118.910228] Could not insert probe at lnet_send+0: -2 [23118.918923] Could not insert probe at lnet_send+0: -2 [23129.157749] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23129.167581] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23139.400469] Could not insert probe at lnet_parse+0: -2 [23139.407883] Could not insert probe at lnet_parse+0: -2 [23149.654342] Could not insert probe at lnet_parse+0: -2 [23149.662307] Could not insert probe at lnet_parse+0: -2 [23159.908881] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23159.916735] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23170.153391] Could not insert probe at lnet_send+0: -2 [23170.161094] Could not insert probe at lnet_send+0: -2 [23180.401002] Could not insert probe at lnet_send+0: -2 [23180.410467] Could not insert probe at lnet_send+0: -2 [23190.653684] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23190.661765] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23200.909433] Could not insert probe at lnet_parse+0: -2 [23200.917420] Could not insert probe at lnet_parse+0: -2 [23211.154737] Could not insert probe at lnet_parse+0: -2 [23211.163793] Could not insert probe at lnet_parse+0: -2 [23221.421750] Could not insert probe at lnet_parse+0: -2 [23221.431096] Could not insert probe at lnet_parse+0: -2 [23231.660025] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23231.668396] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23241.914278] Could not insert probe at lnet_post_send_locked+0: -2 [23241.922035] Could not insert probe at lnet_post_send_locked+0: -2 [23252.160707] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23252.168408] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23262.407619] Could not insert probe at lnet_send+0: -2 [23262.415740] Could not insert probe at lnet_send+0: -2 [23272.657312] Could not insert probe at lnet_parse+0: -2 [23272.666227] Could not insert probe at lnet_parse+0: -2 [23282.911044] Could not insert probe at lnet_parse+0: -2 [23282.918610] Could not insert probe at lnet_parse+0: -2 [23293.163575] Could not insert probe at lnet_send+0: -2 [23293.173132] Could not insert probe at lnet_send+0: -2 [23303.411413] Could not insert probe at lnet_parse+0: -2 [23303.420539] Could not insert probe at lnet_parse+0: -2 [23313.656009] Could not insert probe at lnet_parse+0: -2 [23313.663794] Could not insert probe at lnet_parse+0: -2 [23323.925626] Could not insert probe at lnet_parse+0: -2 [23323.935461] Could not insert probe at lnet_parse+0: -2 [23334.165855] Could not insert probe at lnet_parse+0: -2 [23334.173643] Could not insert probe at lnet_parse+0: -2 [23344.408579] Could not insert probe at lnet_parse+0: -2 [23344.416248] Could not insert probe at lnet_parse+0: -2 [23354.658571] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23354.667481] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23364.911230] Could not insert probe at lnet_parse+0: -2 [23364.918835] Could not insert probe at lnet_parse+0: -2 [23375.166572] Could not insert probe at lnet_parse+0: -2 [23375.174218] Could not insert probe at lnet_parse+0: -2 [23385.415381] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23385.424635] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23395.662882] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23395.670881] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23405.921638] Could not insert probe at lnet_parse+0: -2 [23405.929482] Could not insert probe at lnet_parse+0: -2 [23416.166930] Could not insert probe at lnet_parse+0: -2 [23416.175860] Could not insert probe at lnet_parse+0: -2 [23426.415033] Could not insert probe at lnet_parse+0: -2 [23426.422872] Could not insert probe at lnet_parse+0: -2 [23436.660817] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23436.669698] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23446.919239] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23446.927070] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23457.167649] Could not insert probe at lnet_parse+0: -2 [23457.177378] Could not insert probe at lnet_parse+0: -2 [23467.420541] Could not insert probe at lnet_parse+0: -2 [23467.428820] Could not insert probe at lnet_parse+0: -2 [23477.676226] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23477.687101] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23487.922828] Could not insert probe at lnet_parse+0: -2 [23487.930640] Could not insert probe at lnet_parse+0: -2 [23498.168936] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23498.177990] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23508.420256] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23508.428256] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23518.668798] Could not insert probe at lnet_parse+0: -2 [23518.677716] Could not insert probe at lnet_parse+0: -2 [23528.920866] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23528.929250] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23539.169014] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23539.177824] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23549.421662] Could not insert probe at lnet_parse+0: -2 [23549.429446] Could not insert probe at lnet_parse+0: -2 [23559.665952] Could not insert probe at lnet_parse+0: -2 [23559.673372] Could not insert probe at lnet_parse+0: -2 [23569.921873] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23569.929864] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23580.173134] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23580.181232] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23590.418387] Could not insert probe at lnet_send+0: -2 [23590.426573] Could not insert probe at lnet_send+0: -2 [23600.667138] Could not insert probe at lnet_send+0: -2 [23600.676015] Could not insert probe at lnet_send+0: -2 [23610.919223] Could not insert probe at lnet_parse+0: -2 [23610.927482] Could not insert probe at lnet_parse+0: -2 [23621.173664] Could not insert probe at lnet_parse+0: -2 [23621.182816] Could not insert probe at lnet_parse+0: -2 [23631.425683] Could not insert probe at lnet_post_send_locked+0: -2 [23631.433834] Could not insert probe at lnet_post_send_locked+0: -2 [23641.677202] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23641.686645] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23651.926963] Could not insert probe at lnet_parse+0: -2 [23651.936075] Could not insert probe at lnet_parse+0: -2 [23662.172405] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23662.180379] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23672.424609] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23672.433875] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23682.670931] Could not insert probe at lnet_parse+0: -2 [23682.678693] Could not insert probe at lnet_parse+0: -2 [23692.925989] Could not insert probe at lnet_parse+0: -2 [23692.933708] Could not insert probe at lnet_parse+0: -2 [23703.173992] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23703.182880] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23713.423657] Could not insert probe at lnet_parse+0: -2 [23713.431318] Could not insert probe at lnet_parse+0: -2 [23723.673517] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23723.681753] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23733.925949] Could not insert probe at lnet_send+0: -2 [23733.935461] Could not insert probe at lnet_send+0: -2 [23744.174080] Could not insert probe at lnet_parse+0: -2 [23744.181511] Could not insert probe at lnet_parse+0: -2 [23754.427815] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23754.436987] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23764.675236] Could not insert probe at lnet_post_send_locked+0: -2 [23764.683293] Could not insert probe at lnet_post_send_locked+0: -2 [23774.929421] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23774.937893] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23785.183019] Could not insert probe at lnet_post_send_locked+0: -2 [23785.192275] Could not insert probe at lnet_post_send_locked+0: -2 [23795.431113] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23795.439935] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23805.675883] Could not insert probe at lnet_parse+0: -2 [23805.684486] Could not insert probe at lnet_parse+0: -2 [23815.932343] Could not insert probe at lnet_send+0: -2 [23815.940499] Could not insert probe at lnet_send+0: -2 [23826.179137] Could not insert probe at lnet_send+0: -2 [23826.186747] Could not insert probe at lnet_send+0: -2 [23836.429245] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23836.437044] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23846.678042] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23846.687538] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23856.939816] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23856.949181] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23867.191559] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23867.201498] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23877.428947] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23877.436707] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23887.680348] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23887.688183] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23897.939212] Could not insert probe at lnet_send+0: -2 [23897.948077] Could not insert probe at lnet_send+0: -2 [23908.185583] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23908.195525] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23918.429712] Could not insert probe at lnet_parse+0: -2 [23918.437322] Could not insert probe at lnet_parse+0: -2 [23928.680365] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23928.689913] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23938.936472] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23938.944323] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [23949.189951] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23949.197572] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23959.434792] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23959.443905] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [23969.681875] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23969.689901] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [23979.939251] Could not insert probe at lnet_parse+0: -2 [23979.946740] Could not insert probe at lnet_parse+0: -2 [23990.189627] Could not insert probe at lnet_post_send_locked+0: -2 [23990.197234] Could not insert probe at lnet_post_send_locked+0: -2 [24000.435031] Could not insert probe at lnet_send+0: -2 [24000.442509] Could not insert probe at lnet_send+0: -2 [24010.683650] Could not insert probe at lnet_send+0: -2 [24010.692940] Could not insert probe at lnet_send+0: -2 [24020.934676] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24020.942427] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24031.194509] Could not insert probe at lnet_send+0: -2 [24031.202854] Could not insert probe at lnet_send+0: -2 [24041.435608] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24041.444353] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24051.685437] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24051.693455] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24061.943667] Could not insert probe at lnet_post_send_locked+0: -2 [24061.953041] Could not insert probe at lnet_post_send_locked+0: -2 [24072.191759] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24072.199678] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24082.438130] Could not insert probe at lnet_parse+0: -2 [24082.447957] Could not insert probe at lnet_parse+0: -2 [24092.688782] Could not insert probe at lnet_parse+0: -2 [24092.697049] Could not insert probe at lnet_parse+0: -2 [24102.941122] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24102.950173] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24113.195270] Could not insert probe at lnet_send+0: -2 [24113.205329] Could not insert probe at lnet_send+0: -2 [24123.443808] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24123.451856] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24133.688103] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24133.697686] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24143.940004] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24143.947963] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24154.196527] Could not insert probe at lnet_send+0: -2 [24154.204465] Could not insert probe at lnet_send+0: -2 [24164.439794] Could not insert probe at lnet_parse+0: -2 [24164.448894] Could not insert probe at lnet_parse+0: -2 [24174.692566] Could not insert probe at lnet_send+0: -2 [24174.700225] Could not insert probe at lnet_send+0: -2 [24184.944546] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24184.952822] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24195.193845] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24195.201731] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24205.460293] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24205.468534] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24215.706735] Could not insert probe at lnet_post_send_locked+0: -2 [24215.716254] Could not insert probe at lnet_post_send_locked+0: -2 [24225.953456] Could not insert probe at lnet_parse+0: -2 [24225.964468] Could not insert probe at lnet_parse+0: -2 [24236.197961] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24236.205705] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24246.443842] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24246.452109] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24256.705235] Could not insert probe at lnet_parse+0: -2 [24256.714508] Could not insert probe at lnet_parse+0: -2 [24266.953210] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24266.963092] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24277.206680] Could not insert probe at lnet_parse+0: -2 [24277.214296] Could not insert probe at lnet_parse+0: -2 [24287.454309] Could not insert probe at lnet_send+0: -2 [24287.463920] Could not insert probe at lnet_send+0: -2 [24297.696061] Could not insert probe at lnet_send+0: -2 [24297.703982] Could not insert probe at lnet_send+0: -2 [24307.947254] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24307.955658] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24318.211212] Could not insert probe at lnet_parse+0: -2 [24318.222034] Could not insert probe at lnet_parse+0: -2 [24328.446921] Could not insert probe at lnet_send+0: -2 [24328.456379] Could not insert probe at lnet_send+0: -2 [24338.694921] Could not insert probe at lnet_parse+0: -2 [24338.702659] Could not insert probe at lnet_parse+0: -2 [24348.953427] Could not insert probe at lnet_post_send_locked+0: -2 [24348.961253] Could not insert probe at lnet_post_send_locked+0: -2 [24359.210349] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24359.220308] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24369.459129] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24369.468905] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24379.710293] Could not insert probe at lnet_post_send_locked+0: -2 [24379.720787] Could not insert probe at lnet_post_send_locked+0: -2 [24389.957802] Could not insert probe at lnet_send+0: -2 [24389.966573] Could not insert probe at lnet_send+0: -2 [24400.209658] Could not insert probe at lnet_post_send_locked+0: -2 [24400.219384] Could not insert probe at lnet_post_send_locked+0: -2 [24410.458256] Could not insert probe at lnet_send+0: -2 [24410.467461] Could not insert probe at lnet_send+0: -2 [24420.709598] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24420.718944] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24430.959731] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24430.969475] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24441.211332] Could not insert probe at lnet_parse+0: -2 [24441.221024] Could not insert probe at lnet_parse+0: -2 [24451.450077] Could not insert probe at lnet_parse+0: -2 [24451.458120] Could not insert probe at lnet_parse+0: -2 [24461.710534] Could not insert probe at lnet_parse+0: -2 [24461.719250] Could not insert probe at lnet_parse+0: -2 [24471.961485] Could not insert probe at lnet_post_send_locked+0: -2 [24471.970951] Could not insert probe at lnet_post_send_locked+0: -2 [24482.212062] Could not insert probe at lnet_parse+0: -2 [24482.220507] Could not insert probe at lnet_parse+0: -2 [24492.470298] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24492.479914] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24502.712047] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24502.720989] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24512.970867] Could not insert probe at lnet_send+0: -2 [24512.981656] Could not insert probe at lnet_send+0: -2 [24523.214677] Could not insert probe at lnet_parse+0: -2 [24523.223747] Could not insert probe at lnet_parse+0: -2 [24533.469028] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24533.478457] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24543.714557] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24543.723901] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24553.967467] Could not insert probe at lnet_send+0: -2 [24553.977220] Could not insert probe at lnet_send+0: -2 [24564.214339] Could not insert probe at lnet_send+0: -2 [24564.223339] Could not insert probe at lnet_send+0: -2 [24574.469551] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24574.478867] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24584.714516] Could not insert probe at lnet_parse+0: -2 [24584.722194] Could not insert probe at lnet_parse+0: -2 [24594.967542] Could not insert probe at lnet_send+0: -2 [24594.976785] Could not insert probe at lnet_send+0: -2 [24605.216871] Could not insert probe at lnet_parse+0: -2 [24605.226127] Could not insert probe at lnet_parse+0: -2 [24615.470248] Could not insert probe at lnet_post_send_locked+0: -2 [24615.479327] Could not insert probe at lnet_post_send_locked+0: -2 [24625.720660] Could not insert probe at lnet_parse+0: -2 [24625.729971] Could not insert probe at lnet_parse+0: -2 [24635.965216] Could not insert probe at lnet_parse+0: -2 [24635.974399] Could not insert probe at lnet_parse+0: -2 [24646.219276] Could not insert probe at lnet_parse+0: -2 [24646.228752] Could not insert probe at lnet_parse+0: -2 [24656.468283] Could not insert probe at lnet_parse+0: -2 [24656.476126] Could not insert probe at lnet_parse+0: -2 [24666.720808] Could not insert probe at lnet_post_send_locked+0: -2 [24666.730372] Could not insert probe at lnet_post_send_locked+0: -2 [24676.968586] Could not insert probe at lnet_parse+0: -2 [24676.978080] Could not insert probe at lnet_parse+0: -2 [24687.218292] Could not insert probe at lnet_parse+0: -2 [24687.227322] Could not insert probe at lnet_parse+0: -2 [24697.466665] Could not insert probe at lnet_parse+0: -2 [24697.476037] Could not insert probe at lnet_parse+0: -2 [24707.724294] Could not insert probe at lnet_send+0: -2 [24707.734073] Could not insert probe at lnet_send+0: -2 [24717.968751] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24717.978658] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24728.226324] Could not insert probe at lnet_parse+0: -2 [24728.236020] Could not insert probe at lnet_parse+0: -2 [24738.471082] Could not insert probe at lnet_parse+0: -2 [24738.480171] Could not insert probe at lnet_parse+0: -2 [24748.713486] Could not insert probe at lnet_parse+0: -2 [24748.722779] Could not insert probe at lnet_parse+0: -2 [24758.972084] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24758.982993] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24769.225374] Could not insert probe at lnet_parse+0: -2 [24769.234640] Could not insert probe at lnet_parse+0: -2 [24779.473472] Could not insert probe at lnet_parse+0: -2 [24779.482813] Could not insert probe at lnet_parse+0: -2 [24789.726333] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24789.736355] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24799.974144] Could not insert probe at lnet_parse+0: -2 [24799.983813] Could not insert probe at lnet_parse+0: -2 [24810.227213] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24810.237011] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24820.474007] Could not insert probe at lnet_parse+0: -2 [24820.483658] Could not insert probe at lnet_parse+0: -2 [24830.714561] Could not insert probe at lnet_parse+0: -2 [24830.723894] Could not insert probe at lnet_parse+0: -2 [24840.975722] Could not insert probe at lnet_post_send_locked+0: -2 [24840.985171] Could not insert probe at lnet_post_send_locked+0: -2 [24851.224918] Could not insert probe at lnet_parse+0: -2 [24851.234451] Could not insert probe at lnet_parse+0: -2 [24861.477874] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24861.486974] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24871.726817] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24871.736896] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24881.981846] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24881.990927] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24892.227680] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24892.237189] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24902.482560] Could not insert probe at lnet_parse+0: -2 [24902.491612] Could not insert probe at lnet_parse+0: -2 [24912.727357] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24912.737058] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [24922.985273] Could not insert probe at lnet_parse+0: -2 [24922.993760] Could not insert probe at lnet_parse+0: -2 [24933.231166] Could not insert probe at lnet_parse+0: -2 [24933.241008] Could not insert probe at lnet_parse+0: -2 [24943.469630] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24943.478168] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24953.731079] Could not insert probe at lnet_post_send_locked+0: -2 [24953.740494] Could not insert probe at lnet_post_send_locked+0: -2 [24963.984803] Could not insert probe at lnet_parse+0: -2 [24963.994240] Could not insert probe at lnet_parse+0: -2 [24974.230387] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24974.238513] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [24984.486604] Could not insert probe at lnet_parse+0: -2 [24984.495815] Could not insert probe at lnet_parse+0: -2 [24994.731587] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [24994.741217] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25004.986585] Could not insert probe at lnet_send+0: -2 [25004.995428] Could not insert probe at lnet_send+0: -2 [25015.231887] Could not insert probe at lnet_parse+0: -2 [25015.240831] Could not insert probe at lnet_parse+0: -2 [25025.482192] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25025.491475] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25035.732671] Could not insert probe at lnet_send+0: -2 [25035.741884] Could not insert probe at lnet_send+0: -2 [25045.982417] Could not insert probe at lnet_parse+0: -2 [25045.992807] Could not insert probe at lnet_parse+0: -2 [25056.234420] Could not insert probe at lnet_parse+0: -2 [25056.243777] Could not insert probe at lnet_parse+0: -2 [25066.484109] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25066.494080] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25076.733325] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25076.742498] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25086.986323] Could not insert probe at lnet_post_send_locked+0: -2 [25086.996080] Could not insert probe at lnet_post_send_locked+0: -2 [25097.237905] Could not insert probe at lnet_send+0: -2 [25097.246300] Could not insert probe at lnet_send+0: -2 [25107.485880] Could not insert probe at lnet_send+0: -2 [25107.496697] Could not insert probe at lnet_send+0: -2 [25117.735416] Could not insert probe at lnet_post_send_locked+0: -2 [25117.744857] Could not insert probe at lnet_post_send_locked+0: -2 [25127.986632] Could not insert probe at lnet_post_send_locked+0: -2 [25127.997449] Could not insert probe at lnet_post_send_locked+0: -2 [25138.242822] Could not insert probe at lnet_parse+0: -2 [25138.251603] Could not insert probe at lnet_parse+0: -2 [25148.486135] Could not insert probe at lnet_post_send_locked+0: -2 [25148.495455] Could not insert probe at lnet_post_send_locked+0: -2 [25158.745981] Could not insert probe at lnet_parse+0: -2 [25158.755275] Could not insert probe at lnet_parse+0: -2 [25168.987587] Could not insert probe at lnet_send+0: -2 [25168.997217] Could not insert probe at lnet_send+0: -2 [25179.241418] Could not insert probe at lnet_post_send_locked+0: -2 [25179.250343] Could not insert probe at lnet_post_send_locked+0: -2 [25189.487923] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25189.498904] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25199.736250] Could not insert probe at lnet_post_send_locked+0: -2 [25199.745019] Could not insert probe at lnet_post_send_locked+0: -2 [25209.989108] Could not insert probe at lnet_send+0: -2 [25209.998612] Could not insert probe at lnet_send+0: -2 [25220.244058] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25220.254931] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25230.482020] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25230.490374] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25240.741398] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25240.750895] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25250.992172] Could not insert probe at lnet_parse+0: -2 [25251.001287] Could not insert probe at lnet_parse+0: -2 [25261.243260] Could not insert probe at lnet_parse+0: -2 [25261.252661] Could not insert probe at lnet_parse+0: -2 [25271.492652] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25271.501843] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25281.741021] Could not insert probe at lnet_send+0: -2 [25281.750408] Could not insert probe at lnet_send+0: -2 [25291.996613] Could not insert probe at lnet_parse+0: -2 [25292.006005] Could not insert probe at lnet_parse+0: -2 [25302.243079] Could not insert probe at lnet_parse+0: -2 [25302.252626] Could not insert probe at lnet_parse+0: -2 [25312.490528] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25312.499625] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25322.744979] Could not insert probe at lnet_parse+0: -2 [25322.754205] Could not insert probe at lnet_parse+0: -2 [25332.987721] Could not insert probe at lnet_parse+0: -2 [25332.995470] Could not insert probe at lnet_parse+0: -2 [25343.245447] Could not insert probe at lnet_parse+0: -2 [25343.255223] Could not insert probe at lnet_parse+0: -2 [25353.491424] Could not insert probe at lnet_post_send_locked+0: -2 [25353.499313] Could not insert probe at lnet_post_send_locked+0: -2 [25363.738410] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25363.746583] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25373.992605] Could not insert probe at lnet_parse+0: -2 [25374.001410] Could not insert probe at lnet_parse+0: -2 [25384.246277] Could not insert probe at lnet_send+0: -2 [25384.254432] Could not insert probe at lnet_send+0: -2 [25394.490046] Could not insert probe at lnet_parse+0: -2 [25394.497712] Could not insert probe at lnet_parse+0: -2 [25404.739154] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25404.749199] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25414.990966] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25414.998633] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25425.249817] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25425.258034] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25435.493825] Could not insert probe at lnet_parse+0: -2 [25435.501307] Could not insert probe at lnet_parse+0: -2 [25445.744060] Could not insert probe at lnet_parse+0: -2 [25445.753591] Could not insert probe at lnet_parse+0: -2 [25455.993299] Could not insert probe at lnet_parse+0: -2 [25456.001686] Could not insert probe at lnet_parse+0: -2 [25466.252464] Could not insert probe at lnet_parse+0: -2 [25466.260650] Could not insert probe at lnet_parse+0: -2 [25476.497399] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25476.507116] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25486.742460] Could not insert probe at lnet_send+0: -2 [25486.750260] Could not insert probe at lnet_send+0: -2 [25496.996337] Could not insert probe at lnet_post_send_locked+0: -2 [25497.004779] Could not insert probe at lnet_post_send_locked+0: -2 [25507.250582] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25507.259037] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25517.500901] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25517.508689] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25527.749675] Could not insert probe at lnet_parse+0: -2 [25527.757124] Could not insert probe at lnet_parse+0: -2 [25537.996834] Could not insert probe at lnet_parse+0: -2 [25538.004416] Could not insert probe at lnet_parse+0: -2 [25548.253403] Could not insert probe at lnet_parse+0: -2 [25548.262879] Could not insert probe at lnet_parse+0: -2 [25558.493051] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25558.501140] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25568.743646] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25568.752551] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25578.996045] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25579.003941] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25589.254417] Could not insert probe at lnet_parse+0: -2 [25589.262394] Could not insert probe at lnet_parse+0: -2 [25599.495445] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25599.503416] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25609.749147] Could not insert probe at lnet_post_send_locked+0: -2 [25609.757219] Could not insert probe at lnet_post_send_locked+0: -2 [25620.001424] Could not insert probe at lnet_parse+0: -2 [25620.010607] Could not insert probe at lnet_parse+0: -2 [25630.257128] Could not insert probe at lnet_parse+0: -2 [25630.265051] Could not insert probe at lnet_parse+0: -2 [25640.496715] Could not insert probe at lnet_parse+0: -2 [25640.504327] Could not insert probe at lnet_parse+0: -2 [25650.749612] Could not insert probe at lnet_send+0: -2 [25650.758705] Could not insert probe at lnet_send+0: -2 [25661.000023] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25661.009187] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25671.261080] Could not insert probe at lnet_parse+0: -2 [25671.270943] Could not insert probe at lnet_parse+0: -2 [25681.500459] Could not insert probe at lnet_parse+0: -2 [25681.507923] Could not insert probe at lnet_parse+0: -2 [25691.749137] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25691.758473] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25702.005847] Could not insert probe at lnet_parse+0: -2 [25702.013818] Could not insert probe at lnet_parse+0: -2 [25712.267684] Could not insert probe at lnet_parse+0: -2 [25712.276915] Could not insert probe at lnet_parse+0: -2 [25722.509802] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25722.519346] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25732.754470] Could not insert probe at lnet_parse+0: -2 [25732.762038] Could not insert probe at lnet_parse+0: -2 [25743.012540] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25743.022183] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25753.256934] Could not insert probe at lnet_parse+0: -2 [25753.264829] Could not insert probe at lnet_parse+0: -2 [25763.501258] Could not insert probe at lnet_parse+0: -2 [25763.510047] Could not insert probe at lnet_parse+0: -2 [25773.751258] Could not insert probe at lnet_parse+0: -2 [25773.758917] Could not insert probe at lnet_parse+0: -2 [25784.004698] Could not insert probe at lnet_post_send_locked+0: -2 [25784.013142] Could not insert probe at lnet_post_send_locked+0: -2 [25794.260599] Could not insert probe at lnet_parse+0: -2 [25794.268474] Could not insert probe at lnet_parse+0: -2 [25804.503955] Could not insert probe at lnet_post_send_locked+0: -2 [25804.511696] Could not insert probe at lnet_post_send_locked+0: -2 [25814.753512] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25814.763129] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [25825.009362] Could not insert probe at lnet_post_send_locked+0: -2 [25825.017389] Could not insert probe at lnet_post_send_locked+0: -2 [25835.265519] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25835.274612] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25845.504684] Could not insert probe at lnet_post_send_locked+0: -2 [25845.512229] Could not insert probe at lnet_post_send_locked+0: -2 [25855.757562] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25855.766914] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25866.007757] Could not insert probe at lnet_send+0: -2 [25866.015613] Could not insert probe at lnet_send+0: -2 [25876.265093] Could not insert probe at lnet_send+0: -2 [25876.272734] Could not insert probe at lnet_send+0: -2 [25886.515276] Could not insert probe at lnet_post_send_locked+0: -2 [25886.525627] Could not insert probe at lnet_post_send_locked+0: -2 [25896.762372] Could not insert probe at lnet_parse+0: -2 [25896.770272] Could not insert probe at lnet_parse+0: -2 [25907.010911] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25907.018766] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25917.266241] Could not insert probe at lnet_send+0: -2 [25917.274136] Could not insert probe at lnet_send+0: -2 [25927.518386] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25927.527546] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [25937.761621] Could not insert probe at lnet_parse+0: -2 [25937.769587] Could not insert probe at lnet_parse+0: -2 [25948.010850] Could not insert probe at lnet_send+0: -2 [25948.018377] Could not insert probe at lnet_send+0: -2 [25958.266791] Could not insert probe at lnet_parse+0: -2 [25958.274688] Could not insert probe at lnet_parse+0: -2 [25968.510889] Could not insert probe at lnet_post_send_locked+0: -2 [25968.519133] Could not insert probe at lnet_post_send_locked+0: -2 [25978.760546] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25978.768510] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [25989.012994] Could not insert probe at lnet_parse+0: -2 [25989.021004] Could not insert probe at lnet_parse+0: -2 [25999.269360] Could not insert probe at lnet_post_send_locked+0: -2 [25999.277432] Could not insert probe at lnet_post_send_locked+0: -2 [26009.512637] Could not insert probe at lnet_send+0: -2 [26009.520773] Could not insert probe at lnet_send+0: -2 [26019.760487] Could not insert probe at lnet_parse+0: -2 [26019.768101] Could not insert probe at lnet_parse+0: -2 [26030.016791] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26030.024639] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26040.270377] Could not insert probe at lnet_send+0: -2 [26040.280643] Could not insert probe at lnet_send+0: -2 [26050.512588] Could not insert probe at lnet_send+0: -2 [26050.520330] Could not insert probe at lnet_send+0: -2 [26060.762976] Could not insert probe at lnet_parse+0: -2 [26060.771740] Could not insert probe at lnet_parse+0: -2 [26071.014842] Could not insert probe at lnet_parse+0: -2 [26071.023200] Could not insert probe at lnet_parse+0: -2 [26081.270628] Could not insert probe at lnet_post_send_locked+0: -2 [26081.278655] Could not insert probe at lnet_post_send_locked+0: -2 [26091.513188] Could not insert probe at lnet_send+0: -2 [26091.520918] Could not insert probe at lnet_send+0: -2 [26101.763471] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26101.774039] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26112.019367] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26112.027818] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26122.272729] Could not insert probe at lnet_post_send_locked+0: -2 [26122.281715] Could not insert probe at lnet_post_send_locked+0: -2 [26132.532342] Could not insert probe at lnet_send+0: -2 [26132.541627] Could not insert probe at lnet_send+0: -2 [26142.765888] Could not insert probe at lnet_post_send_locked+0: -2 [26142.774975] Could not insert probe at lnet_post_send_locked+0: -2 [26153.018764] Could not insert probe at lnet_parse+0: -2 [26153.026332] Could not insert probe at lnet_parse+0: -2 [26163.271204] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26163.280092] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26173.518603] Could not insert probe at lnet_post_send_locked+0: -2 [26173.528124] Could not insert probe at lnet_post_send_locked+0: -2 [26183.766749] Could not insert probe at lnet_parse+0: -2 [26183.774453] Could not insert probe at lnet_parse+0: -2 [26194.023381] Could not insert probe at lnet_parse+0: -2 [26194.031024] Could not insert probe at lnet_parse+0: -2 [26204.277017] Could not insert probe at lnet_send+0: -2 [26204.284831] Could not insert probe at lnet_send+0: -2 [26214.544372] Could not insert probe at lnet_post_send_locked+0: -2 [26214.553549] Could not insert probe at lnet_post_send_locked+0: -2 [26224.769661] Could not insert probe at lnet_parse+0: -2 [26224.779516] Could not insert probe at lnet_parse+0: -2 [26235.026455] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26235.034566] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26245.275478] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26245.284360] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26255.521209] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26255.530299] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26265.772294] Could not insert probe at lnet_parse+0: -2 [26265.779785] Could not insert probe at lnet_parse+0: -2 [26276.028845] Could not insert probe at lnet_parse+0: -2 [26276.037190] Could not insert probe at lnet_parse+0: -2 [26286.278837] Could not insert probe at lnet_parse+0: -2 [26286.286636] Could not insert probe at lnet_parse+0: -2 [26296.528310] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26296.538007] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26306.773533] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26306.781625] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26317.028823] Could not insert probe at lnet_parse+0: -2 [26317.036846] Could not insert probe at lnet_parse+0: -2 [26327.282708] Could not insert probe at lnet_parse+0: -2 [26327.290267] Could not insert probe at lnet_parse+0: -2 [26337.529845] Could not insert probe at lnet_parse+0: -2 [26337.537515] Could not insert probe at lnet_parse+0: -2 [26347.778440] Could not insert probe at lnet_parse+0: -2 [26347.786027] Could not insert probe at lnet_parse+0: -2 [26358.027722] Could not insert probe at lnet_parse+0: -2 [26358.035263] Could not insert probe at lnet_parse+0: -2 [26368.279659] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26368.287867] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26378.527171] Could not insert probe at lnet_send+0: -2 [26378.534845] Could not insert probe at lnet_send+0: -2 [26388.778056] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26388.787536] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26399.028415] Could not insert probe at lnet_parse+0: -2 [26399.035953] Could not insert probe at lnet_parse+0: -2 [26409.289204] Could not insert probe at lnet_post_send_locked+0: -2 [26409.297481] Could not insert probe at lnet_post_send_locked+0: -2 [26419.530379] Could not insert probe at lnet_send+0: -2 [26419.538986] Could not insert probe at lnet_send+0: -2 [26429.778970] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26429.788145] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26440.030707] Could not insert probe at lnet_post_send_locked+0: -2 [26440.038590] Could not insert probe at lnet_post_send_locked+0: -2 [26450.283785] Could not insert probe at lnet_parse+0: -2 [26450.293083] Could not insert probe at lnet_parse+0: -2 [26460.535493] Could not insert probe at lnet_parse+0: -2 [26460.543186] Could not insert probe at lnet_parse+0: -2 [26470.781237] Could not insert probe at lnet_parse+0: -2 [26470.791114] Could not insert probe at lnet_parse+0: -2 [26481.031909] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26481.040164] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26491.287463] Could not insert probe at lnet_parse+0: -2 [26491.295718] Could not insert probe at lnet_parse+0: -2 [26501.532153] Could not insert probe at lnet_parse+0: -2 [26501.540041] Could not insert probe at lnet_parse+0: -2 [26511.782691] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26511.790388] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26522.040539] Could not insert probe at lnet_parse+0: -2 [26522.048894] Could not insert probe at lnet_parse+0: -2 [26532.287558] Could not insert probe at lnet_post_send_locked+0: -2 [26532.295276] Could not insert probe at lnet_post_send_locked+0: -2 [26542.536568] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26542.544530] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26552.786691] Could not insert probe at lnet_parse+0: -2 [26552.796113] Could not insert probe at lnet_parse+0: -2 [26563.032857] Could not insert probe at lnet_post_send_locked+0: -2 [26563.041802] Could not insert probe at lnet_post_send_locked+0: -2 [26573.290926] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26573.301301] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26583.536790] Could not insert probe at lnet_parse+0: -2 [26583.545123] Could not insert probe at lnet_parse+0: -2 [26593.785061] Could not insert probe at lnet_parse+0: -2 [26593.794436] Could not insert probe at lnet_parse+0: -2 [26604.036332] Could not insert probe at lnet_parse+0: -2 [26604.045996] Could not insert probe at lnet_parse+0: -2 [26614.291099] Could not insert probe at lnet_parse+0: -2 [26614.299467] Could not insert probe at lnet_parse+0: -2 [26624.537873] Could not insert probe at lnet_parse+0: -2 [26624.546743] Could not insert probe at lnet_parse+0: -2 [26634.786157] Could not insert probe at lnet_parse+0: -2 [26634.793738] Could not insert probe at lnet_parse+0: -2 [26645.038442] Could not insert probe at lnet_post_send_locked+0: -2 [26645.046665] Could not insert probe at lnet_post_send_locked+0: -2 [26655.288854] Could not insert probe at lnet_send+0: -2 [26655.296678] Could not insert probe at lnet_send+0: -2 [26665.545408] Could not insert probe at lnet_parse+0: -2 [26665.553299] Could not insert probe at lnet_parse+0: -2 [26675.790243] Could not insert probe at lnet_parse+0: -2 [26675.797841] Could not insert probe at lnet_parse+0: -2 [26686.041601] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26686.050419] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26696.295273] Could not insert probe at lnet_post_send_locked+0: -2 [26696.304559] Could not insert probe at lnet_post_send_locked+0: -2 [26706.545406] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26706.553853] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26716.800183] Could not insert probe at lnet_parse+0: -2 [26716.809258] Could not insert probe at lnet_parse+0: -2 [26727.041870] Could not insert probe at lnet_parse+0: -2 [26727.049871] Could not insert probe at lnet_parse+0: -2 [26737.297483] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26737.307196] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26747.546829] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26747.554487] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [26757.796629] Could not insert probe at lnet_send+0: -2 [26757.806087] Could not insert probe at lnet_send+0: -2 [26768.048326] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26768.056434] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26778.303573] Could not insert probe at lnet_parse+0: -2 [26778.312887] Could not insert probe at lnet_parse+0: -2 [26788.548915] Could not insert probe at lnet_parse+0: -2 [26788.556576] Could not insert probe at lnet_parse+0: -2 [26798.799449] Could not insert probe at lnet_parse+0: -2 [26798.807644] Could not insert probe at lnet_parse+0: -2 [26809.048596] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26809.057086] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26819.302072] Could not insert probe at lnet_send+0: -2 [26819.310007] Could not insert probe at lnet_send+0: -2 [26829.549931] Could not insert probe at lnet_parse+0: -2 [26829.557824] Could not insert probe at lnet_parse+0: -2 [26839.811794] Could not insert probe at lnet_parse+0: -2 [26839.821298] Could not insert probe at lnet_parse+0: -2 [26850.055537] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26850.065688] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26860.309194] Could not insert probe at lnet_send+0: -2 [26860.319366] Could not insert probe at lnet_send+0: -2 [26870.558227] Could not insert probe at lnet_parse+0: -2 [26870.567256] Could not insert probe at lnet_parse+0: -2 [26880.795884] Could not insert probe at lnet_send+0: -2 [26880.804827] Could not insert probe at lnet_send+0: -2 [26891.048049] Could not insert probe at lnet_post_send_locked+0: -2 [26891.056227] Could not insert probe at lnet_post_send_locked+0: -2 [26901.299310] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26901.307663] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26911.551911] Could not insert probe at lnet_parse+0: -2 [26911.560976] Could not insert probe at lnet_parse+0: -2 [26921.798479] Could not insert probe at lnet_parse+0: -2 [26921.806319] Could not insert probe at lnet_parse+0: -2 [26932.054673] Could not insert probe at lnet_post_send_locked+0: -2 [26932.062848] Could not insert probe at lnet_post_send_locked+0: -2 [26942.310298] Could not insert probe at lnet_send+0: -2 [26942.320867] Could not insert probe at lnet_send+0: -2 [26952.555766] Could not insert probe at lnet_parse+0: -2 [26952.563653] Could not insert probe at lnet_parse+0: -2 [26962.799638] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26962.807800] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26973.051907] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26973.060583] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [26983.301194] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26983.309147] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [26993.555154] Could not insert probe at lnet_send+0: -2 [26993.564191] Could not insert probe at lnet_send+0: -2 [27003.799954] Could not insert probe at lnet_parse+0: -2 [27003.807755] Could not insert probe at lnet_parse+0: -2 [27014.065056] Could not insert probe at lnet_send+0: -2 [27014.073045] Could not insert probe at lnet_send+0: -2 [27024.309186] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27024.318449] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27034.565393] Could not insert probe at lnet_post_send_locked+0: -2 [27034.574878] Could not insert probe at lnet_post_send_locked+0: -2 [27044.811144] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27044.820254] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27055.054110] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27055.063113] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27065.312061] Could not insert probe at lnet_post_send_locked+0: -2 [27065.322255] Could not insert probe at lnet_post_send_locked+0: -2 [27075.567193] Could not insert probe at lnet_send+0: -2 [27075.576385] Could not insert probe at lnet_send+0: -2 [27085.813804] Could not insert probe at lnet_parse+0: -2 [27085.822920] Could not insert probe at lnet_parse+0: -2 [27096.057900] Could not insert probe at lnet_parse+0: -2 [27096.067254] Could not insert probe at lnet_parse+0: -2 [27106.313461] Could not insert probe at lnet_parse+0: -2 [27106.322773] Could not insert probe at lnet_parse+0: -2 [27116.566657] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27116.575769] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27126.813910] Could not insert probe at lnet_parse+0: -2 [27126.823374] Could not insert probe at lnet_parse+0: -2 [27137.061871] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27137.070441] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27147.315115] Could not insert probe at lnet_parse+0: -2 [27147.323210] Could not insert probe at lnet_parse+0: -2 [27157.560924] Could not insert probe at lnet_send+0: -2 [27157.571239] Could not insert probe at lnet_send+0: -2 [27167.806628] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27167.814942] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27178.062370] Could not insert probe at lnet_post_send_locked+0: -2 [27178.070434] Could not insert probe at lnet_post_send_locked+0: -2 [27188.326406] Could not insert probe at lnet_parse+0: -2 [27188.335734] Could not insert probe at lnet_parse+0: -2 [27198.566023] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27198.575140] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27208.807938] Could not insert probe at lnet_post_send_locked+0: -2 [27208.815446] Could not insert probe at lnet_post_send_locked+0: -2 [27219.062352] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27219.070835] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27229.316332] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27229.324374] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27239.566616] Could not insert probe at lnet_parse+0: -2 [27239.574775] Could not insert probe at lnet_parse+0: -2 [27249.808675] Could not insert probe at lnet_post_send_locked+0: -2 [27249.816082] Could not insert probe at lnet_post_send_locked+0: -2 [27260.068089] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27260.077074] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27270.312109] Could not insert probe at lnet_parse+0: -2 [27270.319959] Could not insert probe at lnet_parse+0: -2 [27280.568511] Could not insert probe at lnet_parse+0: -2 [27280.576268] Could not insert probe at lnet_parse+0: -2 [27290.810703] Could not insert probe at lnet_parse+0: -2 [27290.818811] Could not insert probe at lnet_parse+0: -2 [27301.068672] Could not insert probe at lnet_send+0: -2 [27301.076222] Could not insert probe at lnet_send+0: -2 [27311.318412] Could not insert probe at lnet_parse+0: -2 [27311.327612] Could not insert probe at lnet_parse+0: -2 [27321.569341] Could not insert probe at lnet_parse+0: -2 [27321.577011] Could not insert probe at lnet_parse+0: -2 [27331.811756] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27331.821452] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27342.066192] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27342.074264] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27352.322359] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27352.330212] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27362.569817] Could not insert probe at lnet_parse+0: -2 [27362.577551] Could not insert probe at lnet_parse+0: -2 [27372.814578] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27372.824703] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27383.066531] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27383.076357] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27393.317942] Could not insert probe at lnet_send+0: -2 [27393.325910] Could not insert probe at lnet_send+0: -2 [27403.571460] Could not insert probe at lnet_send+0: -2 [27403.581052] Could not insert probe at lnet_send+0: -2 [27413.816302] Could not insert probe at lnet_parse+0: -2 [27413.823978] Could not insert probe at lnet_parse+0: -2 [27424.078681] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27424.088348] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27434.322229] Could not insert probe at lnet_send+0: -2 [27434.330363] Could not insert probe at lnet_send+0: -2 [27444.572497] Could not insert probe at lnet_post_send_locked+0: -2 [27444.580680] Could not insert probe at lnet_post_send_locked+0: -2 [27454.822354] Could not insert probe at lnet_parse+0: -2 [27454.832571] Could not insert probe at lnet_parse+0: -2 [27465.072887] Could not insert probe at lnet_parse+0: -2 [27465.083212] Could not insert probe at lnet_parse+0: -2 [27475.325383] Could not insert probe at lnet_parse+0: -2 [27475.332897] Could not insert probe at lnet_parse+0: -2 [27485.576315] Could not insert probe at lnet_parse+0: -2 [27485.585321] Could not insert probe at lnet_parse+0: -2 [27495.821630] Could not insert probe at lnet_send+0: -2 [27495.829556] Could not insert probe at lnet_send+0: -2 [27506.076094] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27506.084253] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27516.327854] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27516.337541] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27526.578799] Could not insert probe at lnet_parse+0: -2 [27526.586950] Could not insert probe at lnet_parse+0: -2 [27536.821208] Could not insert probe at lnet_post_send_locked+0: -2 [27536.830360] Could not insert probe at lnet_post_send_locked+0: -2 [27547.085680] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27547.095726] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27557.329548] Could not insert probe at lnet_post_send_locked+0: -2 [27557.337310] Could not insert probe at lnet_post_send_locked+0: -2 [27567.582491] Could not insert probe at lnet_post_send_locked+0: -2 [27567.590568] Could not insert probe at lnet_post_send_locked+0: -2 [27577.822951] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27577.832017] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27588.078302] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27588.087246] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27598.326234] Could not insert probe at lnet_send+0: -2 [27598.333806] Could not insert probe at lnet_send+0: -2 [27608.582992] Could not insert probe at lnet_post_send_locked+0: -2 [27608.592161] Could not insert probe at lnet_post_send_locked+0: -2 [27618.835068] Could not insert probe at lnet_post_send_locked+0: -2 [27618.844473] Could not insert probe at lnet_post_send_locked+0: -2 [27629.084488] Could not insert probe at lnet_parse+0: -2 [27629.094224] Could not insert probe at lnet_parse+0: -2 [27639.327932] Could not insert probe at lnet_parse+0: -2 [27639.336617] Could not insert probe at lnet_parse+0: -2 [27649.581997] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27649.591916] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27659.825845] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27659.833686] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27670.084206] Could not insert probe at lnet_parse+0: -2 [27670.093104] Could not insert probe at lnet_parse+0: -2 [27680.330810] Could not insert probe at lnet_parse+0: -2 [27680.340110] Could not insert probe at lnet_parse+0: -2 [27690.584006] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27690.592476] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27700.830057] Could not insert probe at lnet_send+0: -2 [27700.839927] Could not insert probe at lnet_send+0: -2 [27711.080953] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27711.089221] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27721.329643] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27721.337657] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27731.585861] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27731.594033] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27741.841235] Could not insert probe at lnet_parse+0: -2 [27741.850782] Could not insert probe at lnet_parse+0: -2 [27752.083846] Could not insert probe at lnet_parse+0: -2 [27752.092013] Could not insert probe at lnet_parse+0: -2 [27762.337004] Could not insert probe at lnet_parse+0: -2 [27762.345699] Could not insert probe at lnet_parse+0: -2 [27772.589438] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27772.597530] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27782.832016] Could not insert probe at lnet_parse+0: -2 [27782.841657] Could not insert probe at lnet_parse+0: -2 [27793.088452] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27793.096448] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27803.340327] Could not insert probe at lnet_post_send_locked+0: -2 [27803.348801] Could not insert probe at lnet_post_send_locked+0: -2 [27813.586756] Could not insert probe at lnet_parse+0: -2 [27813.595608] Could not insert probe at lnet_parse+0: -2 [27823.831146] Could not insert probe at lnet_post_send_locked+0: -2 [27823.840695] Could not insert probe at lnet_post_send_locked+0: -2 [27834.083909] Could not insert probe at lnet_parse+0: -2 [27834.091989] Could not insert probe at lnet_parse+0: -2 [27844.338497] Could not insert probe at lnet_parse+0: -2 [27844.346275] Could not insert probe at lnet_parse+0: -2 [27854.598964] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27854.608870] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27864.833110] Could not insert probe at lnet_parse+0: -2 [27864.841100] Could not insert probe at lnet_parse+0: -2 [27875.088960] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27875.098080] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27885.341955] Could not insert probe at lnet_parse+0: -2 [27885.351069] Could not insert probe at lnet_parse+0: -2 [27895.592033] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27895.600468] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27905.842002] Could not insert probe at lnet_parse+0: -2 [27905.852242] Could not insert probe at lnet_parse+0: -2 [27916.091858] Could not insert probe at lnet_parse+0: -2 [27916.099313] Could not insert probe at lnet_parse+0: -2 [27926.342952] Could not insert probe at lnet_parse+0: -2 [27926.350769] Could not insert probe at lnet_parse+0: -2 [27936.594075] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27936.604480] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [27946.836173] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27946.845227] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27957.086517] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27957.094720] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [27967.346675] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27967.355767] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27977.595847] Could not insert probe at lnet_parse+0: -2 [27977.603645] Could not insert probe at lnet_parse+0: -2 [27987.840435] Could not insert probe at lnet_parse+0: -2 [27987.848096] Could not insert probe at lnet_parse+0: -2 [27998.092996] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [27998.102709] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28008.347065] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28008.354872] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28018.598328] Could not insert probe at lnet_post_send_locked+0: -2 [28018.606179] Could not insert probe at lnet_post_send_locked+0: -2 [28028.841906] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28028.850712] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28039.092212] Could not insert probe at lnet_send+0: -2 [28039.100905] Could not insert probe at lnet_send+0: -2 [28049.348976] Could not insert probe at lnet_parse+0: -2 [28049.357730] Could not insert probe at lnet_parse+0: -2 [28059.594900] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28059.602808] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28069.841470] Could not insert probe at lnet_parse+0: -2 [28069.849223] Could not insert probe at lnet_parse+0: -2 [28080.096603] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28080.104551] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28090.352673] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28090.361009] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28100.598284] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28100.606323] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28110.843697] Could not insert probe at lnet_parse+0: -2 [28110.852839] Could not insert probe at lnet_parse+0: -2 [28121.098238] Could not insert probe at lnet_parse+0: -2 [28121.106132] Could not insert probe at lnet_parse+0: -2 [28131.355081] Could not insert probe at lnet_parse+0: -2 [28131.364673] Could not insert probe at lnet_parse+0: -2 [28141.598429] Could not insert probe at lnet_parse+0: -2 [28141.606044] Could not insert probe at lnet_parse+0: -2 [28151.844409] Could not insert probe at lnet_send+0: -2 [28151.853832] Could not insert probe at lnet_send+0: -2 [28162.097672] Could not insert probe at lnet_parse+0: -2 [28162.106799] Could not insert probe at lnet_parse+0: -2 [28172.351625] Could not insert probe at lnet_parse+0: -2 [28172.359145] Could not insert probe at lnet_parse+0: -2 [28182.602545] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28182.610607] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28192.851091] Could not insert probe at lnet_send+0: -2 [28192.859024] Could not insert probe at lnet_send+0: -2 [28203.102897] Could not insert probe at lnet_parse+0: -2 [28203.111695] Could not insert probe at lnet_parse+0: -2 [28213.354783] Could not insert probe at lnet_parse+0: -2 [28213.362727] Could not insert probe at lnet_parse+0: -2 [28223.601454] Could not insert probe at lnet_post_send_locked+0: -2 [28223.609208] Could not insert probe at lnet_post_send_locked+0: -2 [28233.848955] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28233.857596] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28244.099905] Could not insert probe at lnet_parse+0: -2 [28244.108078] Could not insert probe at lnet_parse+0: -2 [28254.355610] Could not insert probe at lnet_parse+0: -2 [28254.364318] Could not insert probe at lnet_parse+0: -2 [28264.601515] Could not insert probe at lnet_post_send_locked+0: -2 [28264.609754] Could not insert probe at lnet_post_send_locked+0: -2 [28274.850889] Could not insert probe at lnet_parse+0: -2 [28274.859308] Could not insert probe at lnet_parse+0: -2 [28285.104514] Could not insert probe at lnet_parse+0: -2 [28285.113277] Could not insert probe at lnet_parse+0: -2 [28295.358102] Could not insert probe at lnet_send+0: -2 [28295.368255] Could not insert probe at lnet_send+0: -2 [28305.608256] Could not insert probe at lnet_parse+0: -2 [28305.617455] Could not insert probe at lnet_parse+0: -2 [28315.858792] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28315.866733] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28326.106233] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28326.114927] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28336.360574] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28336.368570] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28346.608656] Could not insert probe at lnet_post_send_locked+0: -2 [28346.618067] Could not insert probe at lnet_post_send_locked+0: -2 [28356.853320] Could not insert probe at lnet_parse+0: -2 [28356.861484] Could not insert probe at lnet_parse+0: -2 [28367.108424] Could not insert probe at lnet_parse+0: -2 [28367.117205] Could not insert probe at lnet_parse+0: -2 [28377.362980] Could not insert probe at lnet_parse+0: -2 [28377.371035] Could not insert probe at lnet_parse+0: -2 [28387.617678] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28387.627986] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28397.855396] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28397.865072] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28408.104908] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28408.114954] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28418.366687] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28418.377594] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28428.609809] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28428.619253] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28438.857619] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28438.867684] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28449.109111] Could not insert probe at lnet_post_send_locked+0: -2 [28449.116967] Could not insert probe at lnet_post_send_locked+0: -2 [28459.361576] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28459.371401] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28469.617957] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28469.625859] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28479.870919] Could not insert probe at lnet_send+0: -2 [28479.879170] Could not insert probe at lnet_send+0: -2 [28490.114055] Could not insert probe at lnet_parse+0: -2 [28490.121652] Could not insert probe at lnet_parse+0: -2 [28500.366280] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28500.375364] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28510.616230] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28510.625609] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28520.865841] Could not insert probe at lnet_parse+0: -2 [28520.873665] Could not insert probe at lnet_parse+0: -2 [28531.111194] Could not insert probe at lnet_post_send_locked+0: -2 [28531.119331] Could not insert probe at lnet_post_send_locked+0: -2 [28541.365845] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28541.375672] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28551.612402] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28551.621299] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28561.861477] Could not insert probe at lnet_post_send_locked+0: -2 [28561.869384] Could not insert probe at lnet_post_send_locked+0: -2 [28572.115149] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28572.124254] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28582.370597] Could not insert probe at lnet_send+0: -2 [28582.379527] Could not insert probe at lnet_send+0: -2 [28592.617087] Could not insert probe at lnet_parse+0: -2 [28592.626245] Could not insert probe at lnet_parse+0: -2 [28602.863837] Could not insert probe at lnet_parse+0: -2 [28602.871519] Could not insert probe at lnet_parse+0: -2 [28613.116409] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28613.124353] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28623.369054] Could not insert probe at lnet_post_send_locked+0: -2 [28623.376590] Could not insert probe at lnet_post_send_locked+0: -2 [28633.624907] Could not insert probe at lnet_parse+0: -2 [28633.633895] Could not insert probe at lnet_parse+0: -2 [28643.866958] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28643.874697] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28654.124963] Could not insert probe at lnet_parse+0: -2 [28654.133638] Could not insert probe at lnet_parse+0: -2 [28664.370115] Could not insert probe at lnet_parse+0: -2 [28664.378415] Could not insert probe at lnet_parse+0: -2 [28674.614876] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28674.624942] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28684.872158] Could not insert probe at lnet_post_send_locked+0: -2 [28684.880204] Could not insert probe at lnet_post_send_locked+0: -2 [28695.118681] Could not insert probe at lnet_post_send_locked+0: -2 [28695.126520] Could not insert probe at lnet_post_send_locked+0: -2 [28705.371387] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28705.379034] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28715.620311] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28715.629273] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28725.869533] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28725.877760] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28736.133074] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28736.141202] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28746.373272] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28746.381565] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28756.624460] Could not insert probe at lnet_post_send_locked+0: -2 [28756.633260] Could not insert probe at lnet_post_send_locked+0: -2 [28766.873346] Could not insert probe at lnet_send+0: -2 [28766.881475] Could not insert probe at lnet_send+0: -2 [28777.125791] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28777.133648] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28787.378232] Could not insert probe at lnet_parse+0: -2 [28787.388986] Could not insert probe at lnet_parse+0: -2 [28797.625970] Could not insert probe at lnet_send+0: -2 [28797.633651] Could not insert probe at lnet_send+0: -2 [28807.857824] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28807.866949] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28818.132902] Could not insert probe at lnet_parse+0: -2 [28818.142023] Could not insert probe at lnet_parse+0: -2 [28828.380368] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28828.389880] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28838.628214] Could not insert probe at lnet_parse+0: -2 [28838.635974] Could not insert probe at lnet_parse+0: -2 [28848.879066] Could not insert probe at lnet_send+0: -2 [28848.886487] Could not insert probe at lnet_send+0: -2 [28859.125094] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28859.133044] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [28869.385697] Could not insert probe at lnet_post_send_locked+0: -2 [28869.395410] Could not insert probe at lnet_post_send_locked+0: -2 [28879.634598] Could not insert probe at lnet_parse+0: -2 [28879.644007] Could not insert probe at lnet_parse+0: -2 [28889.883506] Could not insert probe at lnet_parse+0: -2 [28889.894053] Could not insert probe at lnet_parse+0: -2 [28900.131433] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28900.139643] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28910.388364] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28910.397878] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28920.635246] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28920.644316] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28930.889021] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28930.898696] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [28941.135998] Could not insert probe at lnet_send+0: -2 [28941.144235] Could not insert probe at lnet_send+0: -2 [28951.389562] Could not insert probe at lnet_post_send_locked+0: -2 [28951.400500] Could not insert probe at lnet_post_send_locked+0: -2 [28961.636076] Could not insert probe at lnet_parse+0: -2 [28961.646905] Could not insert probe at lnet_parse+0: -2 [28971.890403] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28971.900724] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [28982.137589] Could not insert probe at lnet_post_send_locked+0: -2 [28982.146467] Could not insert probe at lnet_post_send_locked+0: -2 [28992.388629] Could not insert probe at lnet_parse+0: -2 [28992.397081] Could not insert probe at lnet_parse+0: -2 [29002.633481] Could not insert probe at lnet_send+0: -2 [29002.643856] Could not insert probe at lnet_send+0: -2 [29012.882228] Could not insert probe at lnet_post_send_locked+0: -2 [29012.890372] Could not insert probe at lnet_post_send_locked+0: -2 [29023.135861] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29023.144348] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29033.387374] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29033.395590] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29043.646592] Could not insert probe at lnet_parse+0: -2 [29043.656144] Could not insert probe at lnet_parse+0: -2 [29053.887520] Could not insert probe at lnet_parse+0: -2 [29053.895437] Could not insert probe at lnet_parse+0: -2 [29064.141060] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29064.150463] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29074.387794] Could not insert probe at lnet_post_send_locked+0: -2 [29074.397196] Could not insert probe at lnet_post_send_locked+0: -2 [29084.633566] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29084.641556] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29094.888970] Could not insert probe at lnet_post_send_locked+0: -2 [29094.897292] Could not insert probe at lnet_post_send_locked+0: -2 [29105.136024] Could not insert probe at lnet_parse+0: -2 [29105.144840] Could not insert probe at lnet_parse+0: -2 [29115.386766] Could not insert probe at lnet_send+0: -2 [29115.396031] Could not insert probe at lnet_send+0: -2 [29125.634680] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29125.642748] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29135.888313] Could not insert probe at lnet_parse+0: -2 [29135.895596] Could not insert probe at lnet_parse+0: -2 [29146.135616] Could not insert probe at lnet_post_send_locked+0: -2 [29146.144557] Could not insert probe at lnet_post_send_locked+0: -2 [29156.388861] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29156.397847] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29166.639447] Could not insert probe at lnet_parse+0: -2 [29166.648981] Could not insert probe at lnet_parse+0: -2 [29176.892320] Could not insert probe at lnet_parse+0: -2 [29176.900083] Could not insert probe at lnet_parse+0: -2 [29187.136948] Could not insert probe at lnet_post_send_locked+0: -2 [29187.144605] Could not insert probe at lnet_post_send_locked+0: -2 [29197.396056] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29197.404168] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29207.639390] Could not insert probe at lnet_parse+0: -2 [29207.647877] Could not insert probe at lnet_parse+0: -2 [29217.891651] Could not insert probe at lnet_parse+0: -2 [29217.900618] Could not insert probe at lnet_parse+0: -2 [29228.138168] Could not insert probe at lnet_parse+0: -2 [29228.146462] Could not insert probe at lnet_parse+0: -2 [29238.392963] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29238.400653] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29248.643334] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29248.651067] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29258.893250] Could not insert probe at lnet_parse+0: -2 [29258.902380] Could not insert probe at lnet_parse+0: -2 [29269.139979] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29269.148292] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29279.396211] Could not insert probe at lnet_post_send_locked+0: -2 [29279.405190] Could not insert probe at lnet_post_send_locked+0: -2 [29289.640917] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29289.648686] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29299.897340] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29299.904920] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29310.143860] Could not insert probe at lnet_parse+0: -2 [29310.151438] Could not insert probe at lnet_parse+0: -2 [29320.391870] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29320.402240] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29330.644580] Could not insert probe at lnet_post_send_locked+0: -2 [29330.652406] Could not insert probe at lnet_post_send_locked+0: -2 [29340.896344] Could not insert probe at lnet_parse+0: -2 [29340.904797] Could not insert probe at lnet_parse+0: -2 [29351.146071] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29351.156279] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29361.402988] Could not insert probe at lnet_post_send_locked+0: -2 [29361.410463] Could not insert probe at lnet_post_send_locked+0: -2 [29371.641705] Could not insert probe at lnet_parse+0: -2 [29371.650990] Could not insert probe at lnet_parse+0: -2 [29381.896226] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29381.904353] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29392.147757] Could not insert probe at lnet_parse+0: -2 [29392.155721] Could not insert probe at lnet_parse+0: -2 [29402.403058] Could not insert probe at lnet_send+0: -2 [29402.411126] Could not insert probe at lnet_send+0: -2 [29412.652992] Could not insert probe at lnet_parse+0: -2 [29412.662575] Could not insert probe at lnet_parse+0: -2 [29422.900748] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29422.909042] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29433.150177] Could not insert probe at lnet_parse+0: -2 [29433.158261] Could not insert probe at lnet_parse+0: -2 [29443.402488] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29443.411329] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29453.647485] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29453.657178] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29463.905965] Could not insert probe at lnet_parse+0: -2 [29463.914068] Could not insert probe at lnet_parse+0: -2 [29474.157112] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29474.167050] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29484.401874] Could not insert probe at lnet_parse+0: -2 [29484.410651] Could not insert probe at lnet_parse+0: -2 [29494.658030] Could not insert probe at lnet_parse+0: -2 [29494.665755] Could not insert probe at lnet_parse+0: -2 [29504.905117] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29504.913217] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29515.152635] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29515.160471] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29525.401976] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29525.409902] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29535.654661] Could not insert probe at lnet_send+0: -2 [29535.664483] Could not insert probe at lnet_send+0: -2 [29545.914115] Could not insert probe at lnet_parse+0: -2 [29545.921683] Could not insert probe at lnet_parse+0: -2 [29556.158750] Could not insert probe at lnet_post_send_locked+0: -2 [29556.167049] Could not insert probe at lnet_post_send_locked+0: -2 [29566.401015] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29566.410516] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29576.656260] Could not insert probe at lnet_parse+0: -2 [29576.663980] Could not insert probe at lnet_parse+0: -2 [29586.910292] Could not insert probe at lnet_post_send_locked+0: -2 [29586.918180] Could not insert probe at lnet_post_send_locked+0: -2 [29597.156635] Could not insert probe at lnet_parse+0: -2 [29597.164576] Could not insert probe at lnet_parse+0: -2 [29607.407156] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29607.417528] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29617.658217] Could not insert probe at lnet_post_send_locked+0: -2 [29617.666192] Could not insert probe at lnet_post_send_locked+0: -2 [29627.912367] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29627.921256] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29638.156709] Could not insert probe at lnet_send+0: -2 [29638.164205] Could not insert probe at lnet_send+0: -2 [29648.415758] Could not insert probe at lnet_parse+0: -2 [29648.426509] Could not insert probe at lnet_parse+0: -2 [29658.658980] Could not insert probe at lnet_send+0: -2 [29658.668058] Could not insert probe at lnet_send+0: -2 [29668.915544] Could not insert probe at lnet_parse+0: -2 [29668.925423] Could not insert probe at lnet_parse+0: -2 [29679.155923] Could not insert probe at lnet_parse+0: -2 [29679.163866] Could not insert probe at lnet_parse+0: -2 [29689.410170] Could not insert probe at lnet_parse+0: -2 [29689.418113] Could not insert probe at lnet_parse+0: -2 [29699.655272] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29699.663555] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29709.908915] Could not insert probe at lnet_post_send_locked+0: -2 [29709.918097] Could not insert probe at lnet_post_send_locked+0: -2 [29720.164845] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29720.174358] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29730.411289] Could not insert probe at lnet_parse+0: -2 [29730.418696] Could not insert probe at lnet_parse+0: -2 [29740.661111] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29740.669163] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29750.912950] Could not insert probe at lnet_send+0: -2 [29750.922773] Could not insert probe at lnet_send+0: -2 [29761.160652] Could not insert probe at lnet_send+0: -2 [29761.168956] Could not insert probe at lnet_send+0: -2 [29771.411684] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29771.421462] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29781.660955] Could not insert probe at lnet_parse+0: -2 [29781.668851] Could not insert probe at lnet_parse+0: -2 [29791.914995] Could not insert probe at lnet_send+0: -2 [29791.924275] Could not insert probe at lnet_send+0: -2 [29802.168242] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29802.177836] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29812.412764] Could not insert probe at lnet_post_send_locked+0: -2 [29812.422125] Could not insert probe at lnet_post_send_locked+0: -2 [29822.666006] Could not insert probe at lnet_send+0: -2 [29822.675500] Could not insert probe at lnet_send+0: -2 [29832.916039] Could not insert probe at lnet_parse+0: -2 [29832.924155] Could not insert probe at lnet_parse+0: -2 [29843.164153] Could not insert probe at lnet_parse+0: -2 [29843.173069] Could not insert probe at lnet_parse+0: -2 [29853.422328] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29853.430609] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29863.666640] Could not insert probe at lnet_send+0: -2 [29863.676152] Could not insert probe at lnet_send+0: -2 [29873.917909] Could not insert probe at lnet_send+0: -2 [29873.925917] Could not insert probe at lnet_send+0: -2 [29884.163395] Could not insert probe at lnet_parse+0: -2 [29884.172095] Could not insert probe at lnet_parse+0: -2 [29894.422665] Could not insert probe at lnet_send+0: -2 [29894.432159] Could not insert probe at lnet_send+0: -2 [29904.674341] Could not insert probe at lnet_send+0: -2 [29904.683514] Could not insert probe at lnet_send+0: -2 [29914.923670] Could not insert probe at lnet_parse+0: -2 [29914.933243] Could not insert probe at lnet_parse+0: -2 [29925.169994] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29925.179523] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29935.423761] Could not insert probe at lnet_parse+0: -2 [29935.433043] Could not insert probe at lnet_parse+0: -2 [29945.674680] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29945.684236] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [29955.924051] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29955.932575] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [29966.185581] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29966.195417] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [29976.427079] Could not insert probe at lnet_parse+0: -2 [29976.436639] Could not insert probe at lnet_parse+0: -2 [29986.679184] Could not insert probe at lnet_parse+0: -2 [29986.688753] Could not insert probe at lnet_parse+0: -2 [29996.927135] Could not insert probe at lnet_send+0: -2 [29996.936617] Could not insert probe at lnet_send+0: -2 [30007.180085] Could not insert probe at lnet_send+0: -2 [30007.189550] Could not insert probe at lnet_send+0: -2 [30017.427652] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30017.437066] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30027.682486] Could not insert probe at lnet_send+0: -2 [30027.691480] Could not insert probe at lnet_send+0: -2 [30037.928249] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30037.937845] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30048.173565] Could not insert probe at lnet_parse+0: -2 [30048.181292] Could not insert probe at lnet_parse+0: -2 [30058.428968] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30058.438474] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30068.681937] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30068.690904] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30078.930030] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30078.939499] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30089.175813] Could not insert probe at lnet_post_send_locked+0: -2 [30089.187503] Could not insert probe at lnet_post_send_locked+0: -2 [30099.429602] Could not insert probe at lnet_parse+0: -2 [30099.439080] Could not insert probe at lnet_parse+0: -2 [30109.680672] Could not insert probe at lnet_parse+0: -2 [30109.690996] Could not insert probe at lnet_parse+0: -2 [30119.931677] Could not insert probe at lnet_parse+0: -2 [30119.940917] Could not insert probe at lnet_parse+0: -2 [30130.178727] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30130.188418] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30140.427791] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30140.438236] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30150.681944] Could not insert probe at lnet_parse+0: -2 [30150.691329] Could not insert probe at lnet_parse+0: -2 [30160.934475] Could not insert probe at lnet_parse+0: -2 [30160.943573] Could not insert probe at lnet_parse+0: -2 [30171.180157] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30171.188082] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30181.435866] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30181.445475] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30191.683417] Could not insert probe at lnet_parse+0: -2 [30191.692763] Could not insert probe at lnet_parse+0: -2 [30201.937564] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30201.947192] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30212.185350] Could not insert probe at lnet_parse+0: -2 [30212.194631] Could not insert probe at lnet_parse+0: -2 [30222.436106] Could not insert probe at lnet_post_send_locked+0: -2 [30222.447012] Could not insert probe at lnet_post_send_locked+0: -2 [30232.683981] Could not insert probe at lnet_send+0: -2 [30232.693306] Could not insert probe at lnet_send+0: -2 [30242.955011] Could not insert probe at lnet_send+0: -2 [30242.965819] Could not insert probe at lnet_send+0: -2 [30253.187471] Could not insert probe at lnet_parse+0: -2 [30253.195316] Could not insert probe at lnet_parse+0: -2 [30263.432144] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30263.440649] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30273.683831] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30273.691992] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30283.936458] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30283.944454] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30294.182275] Could not insert probe at lnet_parse+0: -2 [30294.189916] Could not insert probe at lnet_parse+0: -2 [30304.433274] Could not insert probe at lnet_send+0: -2 [30304.441363] Could not insert probe at lnet_send+0: -2 [30314.683405] Could not insert probe at lnet_parse+0: -2 [30314.691839] Could not insert probe at lnet_parse+0: -2 [30324.934389] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30324.944109] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30335.182212] Could not insert probe at lnet_parse+0: -2 [30335.190955] Could not insert probe at lnet_parse+0: -2 [30345.434042] Could not insert probe at lnet_parse+0: -2 [30345.441757] Could not insert probe at lnet_parse+0: -2 [30355.686305] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30355.694064] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30365.939638] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30365.947567] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30376.186257] Could not insert probe at lnet_send+0: -2 [30376.193867] Could not insert probe at lnet_send+0: -2 [30386.441522] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30386.449497] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30396.685687] Could not insert probe at lnet_post_send_locked+0: -2 [30396.694833] Could not insert probe at lnet_post_send_locked+0: -2 [30406.936435] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30406.944071] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30417.187243] Could not insert probe at lnet_post_send_locked+0: -2 [30417.195514] Could not insert probe at lnet_post_send_locked+0: -2 [30427.441624] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30427.449968] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30437.687806] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30437.696680] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30447.937350] Could not insert probe at lnet_parse+0: -2 [30447.946785] Could not insert probe at lnet_parse+0: -2 [30458.194663] Could not insert probe at lnet_send+0: -2 [30458.202120] Could not insert probe at lnet_send+0: -2 [30468.442255] Could not insert probe at lnet_parse+0: -2 [30468.450488] Could not insert probe at lnet_parse+0: -2 [30478.688289] Could not insert probe at lnet_parse+0: -2 [30478.695986] Could not insert probe at lnet_parse+0: -2 [30488.943576] Could not insert probe at lnet_parse+0: -2 [30488.951421] Could not insert probe at lnet_parse+0: -2 [30499.195774] Could not insert probe at lnet_parse+0: -2 [30499.207118] Could not insert probe at lnet_parse+0: -2 [30509.444880] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30509.453239] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30519.693035] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30519.701169] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30529.942332] Could not insert probe at lnet_post_send_locked+0: -2 [30529.952034] Could not insert probe at lnet_post_send_locked+0: -2 [30540.194668] Could not insert probe at lnet_parse+0: -2 [30540.202412] Could not insert probe at lnet_parse+0: -2 [30550.444238] Could not insert probe at lnet_send+0: -2 [30550.453910] Could not insert probe at lnet_send+0: -2 [30560.693397] Could not insert probe at lnet_parse+0: -2 [30560.703094] Could not insert probe at lnet_parse+0: -2 [30570.950015] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30570.959557] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30581.198460] Could not insert probe at lnet_send+0: -2 [30581.207954] Could not insert probe at lnet_send+0: -2 [30591.440055] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30591.449404] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30601.690357] Could not insert probe at lnet_parse+0: -2 [30601.698522] Could not insert probe at lnet_parse+0: -2 [30611.944266] Could not insert probe at lnet_send+0: -2 [30611.952267] Could not insert probe at lnet_send+0: -2 [30622.194334] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30622.203610] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30632.442909] Could not insert probe at lnet_post_send_locked+0: -2 [30632.453493] Could not insert probe at lnet_post_send_locked+0: -2 [30642.690428] Could not insert probe at lnet_parse+0: -2 [30642.698290] Could not insert probe at lnet_parse+0: -2 [30652.943488] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30652.952865] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30663.196610] Could not insert probe at lnet_send+0: -2 [30663.206196] Could not insert probe at lnet_send+0: -2 [30673.454995] Could not insert probe at lnet_parse+0: -2 [30673.464476] Could not insert probe at lnet_parse+0: -2 [30683.703035] Could not insert probe at lnet_send+0: -2 [30683.713971] Could not insert probe at lnet_send+0: -2 [30693.947646] Could not insert probe at lnet_parse+0: -2 [30693.955353] Could not insert probe at lnet_parse+0: -2 [30704.204329] Could not insert probe at lnet_parse+0: -2 [30704.213674] Could not insert probe at lnet_parse+0: -2 [30714.459284] Could not insert probe at lnet_parse+0: -2 [30714.469437] Could not insert probe at lnet_parse+0: -2 [30724.696997] Could not insert probe at lnet_parse+0: -2 [30724.704487] Could not insert probe at lnet_parse+0: -2 [30734.957838] Could not insert probe at lnet_send+0: -2 [30734.967307] Could not insert probe at lnet_send+0: -2 [30745.206574] Could not insert probe at lnet_parse+0: -2 [30745.217325] Could not insert probe at lnet_parse+0: -2 [30755.453648] Could not insert probe at lnet_parse+0: -2 [30755.462721] Could not insert probe at lnet_parse+0: -2 [30765.708595] Could not insert probe at lnet_parse+0: -2 [30765.718115] Could not insert probe at lnet_parse+0: -2 [30775.957430] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30775.967819] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30786.209124] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30786.219013] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30796.448834] Could not insert probe at lnet_parse+0: -2 [30796.457766] Could not insert probe at lnet_parse+0: -2 [30806.711670] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30806.722943] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30816.957967] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30816.967132] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30827.213881] Could not insert probe at lnet_send+0: -2 [30827.223531] Could not insert probe at lnet_send+0: -2 [30837.459284] Could not insert probe at lnet_parse+0: -2 [30837.469001] Could not insert probe at lnet_parse+0: -2 [30847.716598] Could not insert probe at lnet_parse+0: -2 [30847.726146] Could not insert probe at lnet_parse+0: -2 [30857.960484] Could not insert probe at lnet_parse+0: -2 [30857.969822] Could not insert probe at lnet_parse+0: -2 [30868.215059] Could not insert probe at lnet_parse+0: -2 [30868.224038] Could not insert probe at lnet_parse+0: -2 [30878.461123] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30878.470337] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30888.711658] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30888.721503] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30898.960627] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30898.970581] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30909.213226] Could not insert probe at lnet_parse+0: -2 [30909.222699] Could not insert probe at lnet_parse+0: -2 [30919.461199] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30919.472139] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [30929.713826] Could not insert probe at lnet_send+0: -2 [30929.723226] Could not insert probe at lnet_send+0: -2 [30939.962547] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30939.971607] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30950.214313] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30950.224214] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30960.462671] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30960.473614] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [30970.710680] Could not insert probe at lnet_parse+0: -2 [30970.719923] Could not insert probe at lnet_parse+0: -2 [30980.960423] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30980.971302] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [30991.213579] Could not insert probe at lnet_send+0: -2 [30991.222766] Could not insert probe at lnet_send+0: -2 [31001.468523] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31001.478043] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31011.715222] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31011.724597] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31021.971581] Could not insert probe at lnet_post_send_locked+0: -2 [31021.981275] Could not insert probe at lnet_post_send_locked+0: -2 [31032.213947] Could not insert probe at lnet_parse+0: -2 [31032.221282] Could not insert probe at lnet_parse+0: -2 [31042.465938] Could not insert probe at lnet_post_send_locked+0: -2 [31042.473889] Could not insert probe at lnet_post_send_locked+0: -2 [31052.712583] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31052.722304] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31062.972623] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [31062.982522] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [31073.217796] Could not insert probe at lnet_post_send_locked+0: -2 [31073.227140] Could not insert probe at lnet_post_send_locked+0: -2 [31083.472757] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [31083.482266] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [31093.719457] Could not insert probe at lnet_parse+0: -2 [31093.728981] Could not insert probe at lnet_parse+0: -2 [31103.972707] Could not insert probe at lnet_parse+0: -2 [31103.982319] Could not insert probe at lnet_parse+0: -2 [31114.220378] Could not insert probe at lnet_parse+0: -2 [31114.229712] Could not insert probe at lnet_parse+0: -2 [31124.471094] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31124.482083] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31134.720958] Could not insert probe at lnet_post_send_locked+0: -2 [31134.730560] Could not insert probe at lnet_post_send_locked+0: -2 [31144.972596] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31144.981005] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31155.220954] Could not insert probe at lnet_parse+0: -2 [31155.230128] Could not insert probe at lnet_parse+0: -2 [31165.469070] Could not insert probe at lnet_parse+0: -2 [31165.478913] Could not insert probe at lnet_parse+0: -2 [31175.721951] Could not insert probe at lnet_send+0: -2 [31175.731077] Could not insert probe at lnet_send+0: -2 [31185.972750] Could not insert probe at lnet_post_send_locked+0: -2 [31185.982228] Could not insert probe at lnet_post_send_locked+0: -2 [31196.222444] Could not insert probe at lnet_post_send_locked+0: -2 [31196.231837] Could not insert probe at lnet_post_send_locked+0: -2 [31206.472283] Could not insert probe at lnet_parse+0: -2 [31206.481248] Could not insert probe at lnet_parse+0: -2 [31216.725049] Could not insert probe at lnet_parse+0: -2 [31216.734446] Could not insert probe at lnet_parse+0: -2 [31226.972990] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [31226.983994] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [31237.226227] Could not insert probe at lnet_post_send_locked+0: -2 [31237.235315] Could not insert probe at lnet_post_send_locked+0: -2 [31247.473971] Could not insert probe at lnet_post_send_locked+0: -2 [31247.484807] Could not insert probe at lnet_post_send_locked+0: -2 [31257.730768] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31257.739216] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31267.975527] Could not insert probe at lnet_send+0: -2 [31267.984480] Could not insert probe at lnet_send+0: -2 [31278.226546] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31278.234574] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31288.470649] Could not insert probe at lnet_post_send_locked+0: -2 [31288.478452] Could not insert probe at lnet_post_send_locked+0: -2 [31298.719607] Could not insert probe at lnet_post_send_locked+0: -2 [31298.727653] Could not insert probe at lnet_post_send_locked+0: -2 [31308.968293] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31308.978274] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31319.224382] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31319.232439] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31329.468586] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [31329.476848] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [31339.722951] Could not insert probe at lnet_post_send_locked+0: -2 [31339.732414] Could not insert probe at lnet_post_send_locked+0: -2 [31349.970378] Could not insert probe at lnet_parse+0: -2 [31349.977984] Could not insert probe at lnet_parse+0: -2 [31360.224580] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31360.234107] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31370.480423] Could not insert probe at lnet_post_send_locked+0: -2 [31370.489623] Could not insert probe at lnet_post_send_locked+0: -2 [31380.722537] Could not insert probe at lnet_post_send_locked+0: -2 [31380.730410] Could not insert probe at lnet_post_send_locked+0: -2 [31390.974874] Could not insert probe at lnet_parse+0: -2 [31390.984434] Could not insert probe at lnet_parse+0: -2 [31401.233515] Could not insert probe at lnet_parse+0: -2 [31401.241544] Could not insert probe at lnet_parse+0: -2 [31411.476327] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31411.486276] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31421.725754] Could not insert probe at lnet_send+0: -2 [31421.733582] Could not insert probe at lnet_send+0: -2 [31431.974831] Could not insert probe at lnet_parse+0: -2 [31431.982720] Could not insert probe at lnet_parse+0: -2 [31442.229835] Could not insert probe at lnet_post_send_locked+0: -2 [31442.239381] Could not insert probe at lnet_post_send_locked+0: -2 [31452.483527] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [31452.491780] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [31462.728433] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31462.736167] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31472.979115] Could not insert probe at lnet_parse+0: -2 [31472.986886] Could not insert probe at lnet_parse+0: -2 [31483.232061] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31483.239863] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31493.477954] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31493.486134] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31503.726067] Could not insert probe at lnet_parse+0: -2 [31503.733613] Could not insert probe at lnet_parse+0: -2 [31513.976552] Could not insert probe at lnet_parse+0: -2 [31513.986191] Could not insert probe at lnet_parse+0: -2 [31524.236985] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31524.246626] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31534.481981] Could not insert probe at lnet_post_send_locked+0: -2 [31534.489932] Could not insert probe at lnet_post_send_locked+0: -2 [31544.727339] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31544.737687] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31554.978227] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31554.987000] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31565.232027] Could not insert probe at lnet_parse+0: -2 [31565.241206] Could not insert probe at lnet_parse+0: -2 [31575.497060] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31575.506507] Could not insert probe at lnet_return_rx_credits_locked+0: -2 [31585.732461] Could not insert probe at lnet_parse+0: -2 [31585.741907] Could not insert probe at lnet_parse+0: -2 [31595.982808] Could not insert probe at lnet_parse+0: -2 [31595.990354] Could not insert probe at lnet_parse+0: -2 [31606.234961] Could not insert probe at lnet_send+0: -2 [31606.243778] Could not insert probe at lnet_send+0: -2 [31616.479666] Could not insert probe at lnet_parse+0: -2 [31616.488333] Could not insert probe at lnet_parse+0: -2 [31626.734902] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [31626.743778] Could not insert probe at lnet_post_routed_recv_locked+0: -2 [31636.980712] Could not insert probe at lnet_send+0: -2 [31636.989847] Could not insert probe at lnet_send+0: -2 [31647.235870] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31647.243201] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31657.492343] Could not insert probe at lnet_send+0: -2 [31657.501280] Could not insert probe at lnet_send+0: -2 [31667.735344] Could not insert probe at lnet_parse+0: -2 [31667.745483] Could not insert probe at lnet_parse+0: -2 [31677.985275] Could not insert probe at lnet_parse+0: -2 [31677.993566] Could not insert probe at lnet_parse+0: -2 [31688.250711] Could not insert probe at lnet_send+0: -2 [31688.260130] Could not insert probe at lnet_send+0: -2 [31698.490721] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31698.499857] Could not insert probe at lnet_return_tx_credits_locked+0: -2 [31708.737510] Could not insert probe at lnet_parse+0: -2 [31708.746918] Could not insert probe at lnet_parse+0: -2 [31718.985742] Could not insert probe at lnet_parse+0: -2 [31718.993730] Could not insert probe at lnet_parse+0: -2 [31729.247741] Could not insert probe at lnet_send+0: -2 [31729.255529] Could not insert probe at lnet_send+0: -2 [31731.403445] LNet: HW NUMA nodes: 2, HW CPU cores: 10, npartitions: 2 [31731.409413] alg: No test for adler32 (adler32-zlib) [31732.216030] LNet: Using FastReg for registration [31732.227589] LNet: Added LNI BB.BB.ID12@o2ibyy [42/1440/0/0] [31732.230446] LNetError: 13500:0:(o2iblnd_cb.c:2523:kiblnd_passive_connect()) Can't accept conn from AA.AA.3.9ID14ID14@o2ibxx on NA (ib1.8400:0:AA.AA.22.59): bad dst nid AA.AA.ID13@o2ibxx [31732.237404] LNet: Added LNI AA.AA.ID14@o2ibxx [42/6384/0/0] [31732.246742] LNet: 13500:0:(o2iblnd_cb.c:2565:kiblnd_passive_connect()) Can't accept conn from AA.AA.ID15@o2ibxx (version 12): max_frags 257 too large (256 wanted) [31732.754211] LNet: 13500:0:(o2iblnd_cb.c:2565:kiblnd_passive_connect()) Can't accept conn from AA.AA.ID16@o2ibxx (version 12): max_frags 257 too large (256 wanted) [31732.756648] LNet: 13500:0:(o2iblnd_cb.c:2565:kiblnd_passive_connect()) Skipped 31 previous similar messages [31774.216970] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [31774.220211] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [31812.927537] Lustre: DEBUG MARKER: Tue Dec 6 08:50:01 2022 [31826.219100] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds [31826.221044] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (7): c: 0, oc: 0, rc: 42 [31850.219913] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [31850.221672] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [31877.221030] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [31877.224114] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [31884.221339] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds [31884.223273] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (7): c: 0, oc: 0, rc: 42 [31896.221767] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [31896.223628] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 1 previous similar message [31896.225443] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [31896.228025] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 1 previous similar message [31927.222959] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [31927.226110] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 2 previous similar messages [31927.227958] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [31927.230649] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 2 previous similar messages [31965.224436] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds [31965.227796] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 5 previous similar messages [31965.229679] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (7): c: 0, oc: 0, rc: 42 [31965.232458] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 5 previous similar messages [32112.959179] Lustre: DEBUG MARKER: Tue Dec 6 08:55:01 2022 [32412.991620] Lustre: DEBUG MARKER: Tue Dec 6 09:00:01 2022 [32713.043979] Lustre: DEBUG MARKER: Tue Dec 6 09:05:01 2022 [33013.071660] Lustre: DEBUG MARKER: Tue Dec 6 09:10:01 2022 [33313.102678] Lustre: DEBUG MARKER: Tue Dec 6 09:15:01 2022 [33613.133871] Lustre: DEBUG MARKER: Tue Dec 6 09:20:01 2022 [33913.164811] Lustre: DEBUG MARKER: Tue Dec 6 09:25:01 2022 [34213.196819] Lustre: DEBUG MARKER: Tue Dec 6 09:30:01 2022 [34513.229612] Lustre: DEBUG MARKER: Tue Dec 6 09:35:01 2022 [34813.261429] Lustre: DEBUG MARKER: Tue Dec 6 09:40:01 2022 [35113.293442] Lustre: DEBUG MARKER: Tue Dec 6 09:45:01 2022 [35413.323825] Lustre: DEBUG MARKER: Tue Dec 6 09:50:01 2022 [35713.356805] Lustre: DEBUG MARKER: Tue Dec 6 09:55:01 2022 [36013.390707] Lustre: DEBUG MARKER: Tue Dec 6 10:00:01 2022 [36313.446914] Lustre: DEBUG MARKER: Tue Dec 6 10:05:02 2022 [36612.480039] Lustre: DEBUG MARKER: Tue Dec 6 10:10:01 2022 [36912.512274] Lustre: DEBUG MARKER: Tue Dec 6 10:15:01 2022 [37212.543902] Lustre: DEBUG MARKER: Tue Dec 6 10:20:01 2022 [37512.576750] Lustre: DEBUG MARKER: Tue Dec 6 10:25:01 2022 [37812.608370] Lustre: DEBUG MARKER: Tue Dec 6 10:30:01 2022 [38112.642210] Lustre: DEBUG MARKER: Tue Dec 6 10:35:01 2022 [38412.673096] Lustre: DEBUG MARKER: Tue Dec 6 10:40:01 2022 [38712.705170] Lustre: DEBUG MARKER: Tue Dec 6 10:45:01 2022 [39012.735729] Lustre: DEBUG MARKER: Tue Dec 6 10:50:01 2022 [39312.768559] Lustre: DEBUG MARKER: Tue Dec 6 10:55:01 2022 [39612.800310] Lustre: DEBUG MARKER: Tue Dec 6 11:00:01 2022 [39912.855734] Lustre: DEBUG MARKER: Tue Dec 6 11:05:01 2022 [40212.889635] Lustre: DEBUG MARKER: Tue Dec 6 11:10:01 2022 [40512.922133] Lustre: DEBUG MARKER: Tue Dec 6 11:15:01 2022 [40812.952327] Lustre: DEBUG MARKER: Tue Dec 6 11:20:01 2022 [41112.984677] Lustre: DEBUG MARKER: Tue Dec 6 11:25:01 2022 [41413.014062] Lustre: DEBUG MARKER: Tue Dec 6 11:30:01 2022 [41713.046607] Lustre: DEBUG MARKER: Tue Dec 6 11:35:01 2022 [42013.077327] Lustre: DEBUG MARKER: Tue Dec 6 11:40:01 2022 [42313.109843] Lustre: DEBUG MARKER: Tue Dec 6 11:45:01 2022 [42613.139491] Lustre: DEBUG MARKER: Tue Dec 6 11:50:01 2022 [42913.172874] Lustre: DEBUG MARKER: Tue Dec 6 11:55:01 2022 [43213.209620] Lustre: DEBUG MARKER: Tue Dec 6 12:00:01 2022 [43513.263777] Lustre: DEBUG MARKER: Tue Dec 6 12:05:01 2022 [43813.297356] Lustre: DEBUG MARKER: Tue Dec 6 12:10:01 2022 [44113.332814] Lustre: DEBUG MARKER: Tue Dec 6 12:15:01 2022 [44413.367040] Lustre: DEBUG MARKER: Tue Dec 6 12:20:01 2022 [44713.400139] Lustre: DEBUG MARKER: Tue Dec 6 12:25:01 2022 [45013.431375] Lustre: DEBUG MARKER: Tue Dec 6 12:30:01 2022 [45313.464555] Lustre: DEBUG MARKER: Tue Dec 6 12:35:01 2022 [45613.494720] Lustre: DEBUG MARKER: Tue Dec 6 12:40:01 2022 [45913.527039] Lustre: DEBUG MARKER: Tue Dec 6 12:45:01 2022 [46213.559625] Lustre: DEBUG MARKER: Tue Dec 6 12:50:01 2022 [46513.593363] Lustre: DEBUG MARKER: Tue Dec 6 12:55:01 2022 [46813.627809] Lustre: DEBUG MARKER: Tue Dec 6 13:00:01 2022 [47113.684803] Lustre: DEBUG MARKER: Tue Dec 6 13:05:01 2022 [47413.717290] Lustre: DEBUG MARKER: Tue Dec 6 13:10:01 2022 [47713.750547] Lustre: DEBUG MARKER: Tue Dec 6 13:15:01 2022 [48013.782213] Lustre: DEBUG MARKER: Tue Dec 6 13:20:01 2022 [48313.816080] Lustre: DEBUG MARKER: Tue Dec 6 13:25:01 2022 [48613.847188] Lustre: DEBUG MARKER: Tue Dec 6 13:30:01 2022 [48913.879015] Lustre: DEBUG MARKER: Tue Dec 6 13:35:01 2022 [49213.912214] Lustre: DEBUG MARKER: Tue Dec 6 13:40:01 2022 [49513.946493] Lustre: DEBUG MARKER: Tue Dec 6 13:45:01 2022 [49812.980406] Lustre: DEBUG MARKER: Tue Dec 6 13:50:01 2022 [50113.014348] Lustre: DEBUG MARKER: Tue Dec 6 13:55:01 2022 [50413.049388] Lustre: DEBUG MARKER: Tue Dec 6 14:00:01 2022 [50713.107370] Lustre: DEBUG MARKER: Tue Dec 6 14:05:01 2022 [51013.139725] Lustre: DEBUG MARKER: Tue Dec 6 14:10:01 2022 [51313.174160] Lustre: DEBUG MARKER: Tue Dec 6 14:15:01 2022 [51613.208048] Lustre: DEBUG MARKER: Tue Dec 6 14:20:01 2022 [51913.240611] Lustre: DEBUG MARKER: Tue Dec 6 14:25:01 2022 [52213.271091] Lustre: DEBUG MARKER: Tue Dec 6 14:30:01 2022 [52513.303960] Lustre: DEBUG MARKER: Tue Dec 6 14:35:01 2022 [52813.337315] Lustre: DEBUG MARKER: Tue Dec 6 14:40:01 2022 [53113.370568] Lustre: DEBUG MARKER: Tue Dec 6 14:45:01 2022 [53413.405388] Lustre: DEBUG MARKER: Tue Dec 6 14:50:01 2022 [53713.437501] Lustre: DEBUG MARKER: Tue Dec 6 14:55:01 2022 [54013.470504] Lustre: DEBUG MARKER: Tue Dec 6 15:00:01 2022 [54313.525134] Lustre: DEBUG MARKER: Tue Dec 6 15:05:01 2022 [54613.556058] Lustre: DEBUG MARKER: Tue Dec 6 15:10:01 2022 [54913.588405] Lustre: DEBUG MARKER: Tue Dec 6 15:15:01 2022 [55213.621102] Lustre: DEBUG MARKER: Tue Dec 6 15:20:01 2022 [55513.655485] Lustre: DEBUG MARKER: Tue Dec 6 15:25:01 2022 [55813.686894] Lustre: DEBUG MARKER: Tue Dec 6 15:30:01 2022 [56113.719962] Lustre: DEBUG MARKER: Tue Dec 6 15:35:01 2022 [56413.752205] Lustre: DEBUG MARKER: Tue Dec 6 15:40:01 2022 [56713.786778] Lustre: DEBUG MARKER: Tue Dec 6 15:45:01 2022 [57013.817808] Lustre: DEBUG MARKER: Tue Dec 6 15:50:01 2022 [57313.850578] Lustre: DEBUG MARKER: Tue Dec 6 15:55:01 2022 [57613.885279] Lustre: DEBUG MARKER: Tue Dec 6 16:00:01 2022 [57913.939166] Lustre: DEBUG MARKER: Tue Dec 6 16:05:01 2022 [58213.970858] Lustre: DEBUG MARKER: Tue Dec 6 16:10:01 2022 [58514.003109] Lustre: DEBUG MARKER: Tue Dec 6 16:15:01 2022 [58814.030670] Lustre: DEBUG MARKER: Tue Dec 6 16:20:01 2022 [59114.061745] Lustre: DEBUG MARKER: Tue Dec 6 16:25:01 2022 [59414.091840] Lustre: DEBUG MARKER: Tue Dec 6 16:30:01 2022 [59714.122841] Lustre: DEBUG MARKER: Tue Dec 6 16:35:01 2022 [60014.151228] Lustre: DEBUG MARKER: Tue Dec 6 16:40:01 2022 [60314.184170] Lustre: DEBUG MARKER: Tue Dec 6 16:45:01 2022 [60614.217019] Lustre: DEBUG MARKER: Tue Dec 6 16:50:01 2022 [60914.251340] Lustre: DEBUG MARKER: Tue Dec 6 16:55:01 2022 [61214.287845] Lustre: DEBUG MARKER: Tue Dec 6 17:00:01 2022 [61514.344760] Lustre: DEBUG MARKER: Tue Dec 6 17:05:01 2022 [61814.376532] Lustre: DEBUG MARKER: Tue Dec 6 17:10:01 2022 [62114.410128] Lustre: DEBUG MARKER: Tue Dec 6 17:15:01 2022 [62414.442907] Lustre: DEBUG MARKER: Tue Dec 6 17:20:01 2022 [62714.476975] Lustre: DEBUG MARKER: Tue Dec 6 17:25:02 2022 [63013.510509] Lustre: DEBUG MARKER: Tue Dec 6 17:30:01 2022 [63313.542474] Lustre: DEBUG MARKER: Tue Dec 6 17:35:01 2022 [63613.576670] Lustre: DEBUG MARKER: Tue Dec 6 17:40:01 2022 [63913.609457] Lustre: DEBUG MARKER: Tue Dec 6 17:45:01 2022 [64213.642648] Lustre: DEBUG MARKER: Tue Dec 6 17:50:01 2022 [64513.674199] Lustre: DEBUG MARKER: Tue Dec 6 17:55:01 2022 [64813.709667] Lustre: DEBUG MARKER: Tue Dec 6 18:00:01 2022 [65113.766597] Lustre: DEBUG MARKER: Tue Dec 6 18:05:01 2022 [65413.798612] Lustre: DEBUG MARKER: Tue Dec 6 18:10:01 2022 [65713.831776] Lustre: DEBUG MARKER: Tue Dec 6 18:15:01 2022 [66013.863380] Lustre: DEBUG MARKER: Tue Dec 6 18:20:01 2022 [66313.895636] Lustre: DEBUG MARKER: Tue Dec 6 18:25:01 2022 [66613.926044] Lustre: DEBUG MARKER: Tue Dec 6 18:30:01 2022 [66913.958840] Lustre: DEBUG MARKER: Tue Dec 6 18:35:01 2022 [67213.990358] Lustre: DEBUG MARKER: Tue Dec 6 18:40:01 2022 [67514.023228] Lustre: DEBUG MARKER: Tue Dec 6 18:45:01 2022 [67814.057681] Lustre: DEBUG MARKER: Tue Dec 6 18:50:01 2022 [68114.089507] Lustre: DEBUG MARKER: Tue Dec 6 18:55:01 2022 [68414.122806] Lustre: DEBUG MARKER: Tue Dec 6 19:00:01 2022 [68714.179274] Lustre: DEBUG MARKER: Tue Dec 6 19:05:01 2022 [69014.209875] Lustre: DEBUG MARKER: Tue Dec 6 19:10:01 2022 [69314.241899] Lustre: DEBUG MARKER: Tue Dec 6 19:15:01 2022 [69614.272241] Lustre: DEBUG MARKER: Tue Dec 6 19:20:01 2022 [69914.304770] Lustre: DEBUG MARKER: Tue Dec 6 19:25:01 2022 [70214.338946] Lustre: DEBUG MARKER: Tue Dec 6 19:30:01 2022 [70514.375399] Lustre: DEBUG MARKER: Tue Dec 6 19:35:01 2022 [70814.409178] Lustre: DEBUG MARKER: Tue Dec 6 19:40:01 2022 [71114.440489] Lustre: DEBUG MARKER: Tue Dec 6 19:45:01 2022 [71414.472459] Lustre: DEBUG MARKER: Tue Dec 6 19:50:01 2022 [71714.505001] Lustre: DEBUG MARKER: Tue Dec 6 19:55:01 2022 [72014.538849] Lustre: DEBUG MARKER: Tue Dec 6 20:00:01 2022 [72314.594502] Lustre: DEBUG MARKER: Tue Dec 6 20:05:01 2022 [72614.626460] Lustre: DEBUG MARKER: Tue Dec 6 20:10:01 2022 [72914.658718] Lustre: DEBUG MARKER: Tue Dec 6 20:15:01 2022 [73214.689486] Lustre: DEBUG MARKER: Tue Dec 6 20:20:01 2022 [73514.724312] Lustre: DEBUG MARKER: Tue Dec 6 20:25:01 2022 [73814.756789] Lustre: DEBUG MARKER: Tue Dec 6 20:30:01 2022 [74114.789421] Lustre: DEBUG MARKER: Tue Dec 6 20:35:01 2022 [74414.821524] Lustre: DEBUG MARKER: Tue Dec 6 20:40:01 2022 [74714.853092] Lustre: DEBUG MARKER: Tue Dec 6 20:45:01 2022 [75014.885463] Lustre: DEBUG MARKER: Tue Dec 6 20:50:01 2022 [75314.917399] Lustre: DEBUG MARKER: Tue Dec 6 20:55:01 2022 [75614.950678] Lustre: DEBUG MARKER: Tue Dec 6 21:00:01 2022 [75914.006071] Lustre: DEBUG MARKER: Tue Dec 6 21:05:01 2022 [76214.036157] Lustre: DEBUG MARKER: Tue Dec 6 21:10:01 2022 [76514.068570] Lustre: DEBUG MARKER: Tue Dec 6 21:15:01 2022 [76814.100451] Lustre: DEBUG MARKER: Tue Dec 6 21:20:01 2022 [77114.133945] Lustre: DEBUG MARKER: Tue Dec 6 21:25:01 2022 [77414.165704] Lustre: DEBUG MARKER: Tue Dec 6 21:30:01 2022 [77714.198492] Lustre: DEBUG MARKER: Tue Dec 6 21:35:01 2022 [78014.229611] Lustre: DEBUG MARKER: Tue Dec 6 21:40:01 2022 [78314.262027] Lustre: DEBUG MARKER: Tue Dec 6 21:45:01 2022 [78614.293109] Lustre: DEBUG MARKER: Tue Dec 6 21:50:01 2022 [78914.325988] Lustre: DEBUG MARKER: Tue Dec 6 21:55:01 2022 [79214.363043] Lustre: DEBUG MARKER: Tue Dec 6 22:00:01 2022 [79514.418683] Lustre: DEBUG MARKER: Tue Dec 6 22:05:01 2022 [79814.449677] Lustre: DEBUG MARKER: Tue Dec 6 22:10:01 2022 [80114.483447] Lustre: DEBUG MARKER: Tue Dec 6 22:15:01 2022 [80414.514683] Lustre: DEBUG MARKER: Tue Dec 6 22:20:01 2022 [80714.547426] Lustre: DEBUG MARKER: Tue Dec 6 22:25:01 2022 [81014.578755] Lustre: DEBUG MARKER: Tue Dec 6 22:30:01 2022 [81314.611790] Lustre: DEBUG MARKER: Tue Dec 6 22:35:01 2022 [81614.643831] Lustre: DEBUG MARKER: Tue Dec 6 22:40:01 2022 [81914.676609] Lustre: DEBUG MARKER: Tue Dec 6 22:45:01 2022 [82214.709530] Lustre: DEBUG MARKER: Tue Dec 6 22:50:01 2022 [82514.742563] Lustre: DEBUG MARKER: Tue Dec 6 22:55:01 2022 [82814.776488] Lustre: DEBUG MARKER: Tue Dec 6 23:00:01 2022 [83114.832342] Lustre: DEBUG MARKER: Tue Dec 6 23:05:01 2022 [83414.865127] Lustre: DEBUG MARKER: Tue Dec 6 23:10:01 2022 [83714.898059] Lustre: DEBUG MARKER: Tue Dec 6 23:15:01 2022 [84014.930038] Lustre: DEBUG MARKER: Tue Dec 6 23:20:01 2022 [84314.962363] Lustre: DEBUG MARKER: Tue Dec 6 23:25:01 2022 [84614.995326] Lustre: DEBUG MARKER: Tue Dec 6 23:30:01 2022 [84915.029842] Lustre: DEBUG MARKER: Tue Dec 6 23:35:01 2022 [85215.062821] Lustre: DEBUG MARKER: Tue Dec 6 23:40:01 2022 [85515.095841] Lustre: DEBUG MARKER: Tue Dec 6 23:45:01 2022 [85815.127514] Lustre: DEBUG MARKER: Tue Dec 6 23:50:01 2022 [86114.408454] Lustre: DEBUG MARKER: Tue Dec 6 23:55:01 2022 [86414.442421] Lustre: DEBUG MARKER: Wed Dec 7 00:00:01 2022 [86714.500538] Lustre: DEBUG MARKER: Wed Dec 7 00:05:01 2022 [87014.532043] Lustre: DEBUG MARKER: Wed Dec 7 00:10:01 2022 [87314.564411] Lustre: DEBUG MARKER: Wed Dec 7 00:15:01 2022 [87614.596283] Lustre: DEBUG MARKER: Wed Dec 7 00:20:01 2022 [87914.629313] Lustre: DEBUG MARKER: Wed Dec 7 00:25:01 2022 [88214.660369] Lustre: DEBUG MARKER: Wed Dec 7 00:30:01 2022 [88514.692753] Lustre: DEBUG MARKER: Wed Dec 7 00:35:01 2022 [88814.722250] Lustre: DEBUG MARKER: Wed Dec 7 00:40:01 2022 [89114.755054] Lustre: DEBUG MARKER: Wed Dec 7 00:45:01 2022 [89414.787929] Lustre: DEBUG MARKER: Wed Dec 7 00:50:01 2022 [89714.819936] Lustre: DEBUG MARKER: Wed Dec 7 00:55:01 2022 [90014.854122] Lustre: DEBUG MARKER: Wed Dec 7 01:00:01 2022 [90314.911413] Lustre: DEBUG MARKER: Wed Dec 7 01:05:01 2022 [90614.943411] Lustre: DEBUG MARKER: Wed Dec 7 01:10:01 2022 [90914.976772] Lustre: DEBUG MARKER: Wed Dec 7 01:15:01 2022 [91215.007803] Lustre: DEBUG MARKER: Wed Dec 7 01:20:01 2022 [91515.040336] Lustre: DEBUG MARKER: Wed Dec 7 01:25:01 2022 [91815.072482] Lustre: DEBUG MARKER: Wed Dec 7 01:30:01 2022 [92115.105548] Lustre: DEBUG MARKER: Wed Dec 7 01:35:01 2022 [92415.138935] Lustre: DEBUG MARKER: Wed Dec 7 01:40:01 2022 [92715.171564] Lustre: DEBUG MARKER: Wed Dec 7 01:45:01 2022 [93015.203008] Lustre: DEBUG MARKER: Wed Dec 7 01:50:01 2022 [93315.236669] Lustre: DEBUG MARKER: Wed Dec 7 01:55:01 2022 [93615.270271] Lustre: DEBUG MARKER: Wed Dec 7 02:00:01 2022 [93915.328723] Lustre: DEBUG MARKER: Wed Dec 7 02:05:01 2022 [94215.358734] Lustre: DEBUG MARKER: Wed Dec 7 02:10:01 2022 [94515.391291] Lustre: DEBUG MARKER: Wed Dec 7 02:15:01 2022 [94815.422922] Lustre: DEBUG MARKER: Wed Dec 7 02:20:01 2022 [95115.455744] Lustre: DEBUG MARKER: Wed Dec 7 02:25:01 2022 [95415.487825] Lustre: DEBUG MARKER: Wed Dec 7 02:30:01 2022 [95715.521655] Lustre: DEBUG MARKER: Wed Dec 7 02:35:01 2022 [96015.553617] Lustre: DEBUG MARKER: Wed Dec 7 02:40:01 2022 [96315.586710] Lustre: DEBUG MARKER: Wed Dec 7 02:45:01 2022 [96615.616145] Lustre: DEBUG MARKER: Wed Dec 7 02:50:01 2022 [96915.649703] Lustre: DEBUG MARKER: Wed Dec 7 02:55:01 2022 [97215.685282] Lustre: DEBUG MARKER: Wed Dec 7 03:00:01 2022 [97515.739722] Lustre: DEBUG MARKER: Wed Dec 7 03:05:01 2022 [97815.770922] Lustre: DEBUG MARKER: Wed Dec 7 03:10:01 2022 [98115.801853] Lustre: DEBUG MARKER: Wed Dec 7 03:15:01 2022 [98415.831270] Lustre: DEBUG MARKER: Wed Dec 7 03:20:01 2022 [98715.862016] Lustre: DEBUG MARKER: Wed Dec 7 03:25:01 2022 [99014.891971] Lustre: DEBUG MARKER: Wed Dec 7 03:30:01 2022 [99314.922634] Lustre: DEBUG MARKER: Wed Dec 7 03:35:01 2022 [99614.952612] Lustre: DEBUG MARKER: Wed Dec 7 03:40:01 2022 [99914.984985] Lustre: DEBUG MARKER: Wed Dec 7 03:45:01 2022 [100215.016036] Lustre: DEBUG MARKER: Wed Dec 7 03:50:01 2022 [100515.049091] Lustre: DEBUG MARKER: Wed Dec 7 03:55:01 2022 [100815.082057] Lustre: DEBUG MARKER: Wed Dec 7 04:00:01 2022 [101115.138615] Lustre: DEBUG MARKER: Wed Dec 7 04:05:01 2022 [101415.169679] Lustre: DEBUG MARKER: Wed Dec 7 04:10:01 2022 [101715.202088] Lustre: DEBUG MARKER: Wed Dec 7 04:15:01 2022 [102015.234801] Lustre: DEBUG MARKER: Wed Dec 7 04:20:01 2022 [102315.267776] Lustre: DEBUG MARKER: Wed Dec 7 04:25:01 2022 [102615.299348] Lustre: DEBUG MARKER: Wed Dec 7 04:30:01 2022 [102915.331364] Lustre: DEBUG MARKER: Wed Dec 7 04:35:01 2022 [103215.363042] Lustre: DEBUG MARKER: Wed Dec 7 04:40:01 2022 [103515.395855] Lustre: DEBUG MARKER: Wed Dec 7 04:45:01 2022 [103815.427011] Lustre: DEBUG MARKER: Wed Dec 7 04:50:01 2022 [104115.459332] Lustre: DEBUG MARKER: Wed Dec 7 04:55:01 2022 [104415.491110] Lustre: DEBUG MARKER: Wed Dec 7 05:00:01 2022 [104715.546307] Lustre: DEBUG MARKER: Wed Dec 7 05:05:01 2022 [105015.578316] Lustre: DEBUG MARKER: Wed Dec 7 05:10:01 2022 [105315.611105] Lustre: DEBUG MARKER: Wed Dec 7 05:15:01 2022 [105615.643062] Lustre: DEBUG MARKER: Wed Dec 7 05:20:01 2022 [105915.676585] Lustre: DEBUG MARKER: Wed Dec 7 05:25:01 2022 [106215.707354] Lustre: DEBUG MARKER: Wed Dec 7 05:30:01 2022 [106515.740061] Lustre: DEBUG MARKER: Wed Dec 7 05:35:01 2022 [106815.771257] Lustre: DEBUG MARKER: Wed Dec 7 05:40:01 2022 [107115.803644] Lustre: DEBUG MARKER: Wed Dec 7 05:45:01 2022 [107415.835475] Lustre: DEBUG MARKER: Wed Dec 7 05:50:01 2022 [107715.868363] Lustre: DEBUG MARKER: Wed Dec 7 05:55:01 2022 [108015.900671] Lustre: DEBUG MARKER: Wed Dec 7 06:00:01 2022 [108315.957785] Lustre: DEBUG MARKER: Wed Dec 7 06:05:01 2022 [108615.989383] Lustre: DEBUG MARKER: Wed Dec 7 06:10:01 2022 [108916.024402] Lustre: DEBUG MARKER: Wed Dec 7 06:15:01 2022 [109216.056281] Lustre: DEBUG MARKER: Wed Dec 7 06:20:01 2022 [109516.089307] Lustre: DEBUG MARKER: Wed Dec 7 06:25:01 2022 [109816.121182] Lustre: DEBUG MARKER: Wed Dec 7 06:30:01 2022 [110116.152714] Lustre: DEBUG MARKER: Wed Dec 7 06:35:01 2022 [110416.184397] Lustre: DEBUG MARKER: Wed Dec 7 06:40:01 2022 [110716.217659] Lustre: DEBUG MARKER: Wed Dec 7 06:45:01 2022 [111016.250075] Lustre: DEBUG MARKER: Wed Dec 7 06:50:01 2022 [111316.282601] Lustre: DEBUG MARKER: Wed Dec 7 06:55:01 2022 [111616.315774] Lustre: DEBUG MARKER: Wed Dec 7 07:00:01 2022 [111916.372214] Lustre: DEBUG MARKER: Wed Dec 7 07:05:01 2022 [112215.401970] Lustre: DEBUG MARKER: Wed Dec 7 07:10:01 2022 [112515.435384] Lustre: DEBUG MARKER: Wed Dec 7 07:15:01 2022 [112815.467375] Lustre: DEBUG MARKER: Wed Dec 7 07:20:01 2022 [113115.501321] Lustre: DEBUG MARKER: Wed Dec 7 07:25:01 2022 [113415.535034] Lustre: DEBUG MARKER: Wed Dec 7 07:30:01 2022 [113715.569229] Lustre: DEBUG MARKER: Wed Dec 7 07:35:01 2022 [114015.600714] Lustre: DEBUG MARKER: Wed Dec 7 07:40:01 2022 [114315.634223] Lustre: DEBUG MARKER: Wed Dec 7 07:45:01 2022 [114615.663989] Lustre: DEBUG MARKER: Wed Dec 7 07:50:01 2022 [114915.697047] Lustre: DEBUG MARKER: Wed Dec 7 07:55:01 2022 [115215.730049] Lustre: DEBUG MARKER: Wed Dec 7 08:00:01 2022 [115515.790610] Lustre: DEBUG MARKER: Wed Dec 7 08:05:01 2022 [115815.821706] Lustre: DEBUG MARKER: Wed Dec 7 08:10:01 2022 [116115.855076] Lustre: DEBUG MARKER: Wed Dec 7 08:15:01 2022 [116415.885655] Lustre: DEBUG MARKER: Wed Dec 7 08:20:01 2022 [116715.918425] Lustre: DEBUG MARKER: Wed Dec 7 08:25:01 2022 [117015.949421] Lustre: DEBUG MARKER: Wed Dec 7 08:30:01 2022 [117315.981906] Lustre: DEBUG MARKER: Wed Dec 7 08:35:01 2022 [117616.012195] Lustre: DEBUG MARKER: Wed Dec 7 08:40:01 2022 [117916.044278] Lustre: DEBUG MARKER: Wed Dec 7 08:45:01 2022 [118216.075473] Lustre: DEBUG MARKER: Wed Dec 7 08:50:01 2022 [118516.107964] Lustre: DEBUG MARKER: Wed Dec 7 08:55:01 2022 [118816.144026] Lustre: DEBUG MARKER: Wed Dec 7 09:00:01 2022 [119116.202172] Lustre: DEBUG MARKER: Wed Dec 7 09:05:01 2022 [119416.235963] Lustre: DEBUG MARKER: Wed Dec 7 09:10:01 2022 [119716.268300] Lustre: DEBUG MARKER: Wed Dec 7 09:15:01 2022 [120016.300195] Lustre: DEBUG MARKER: Wed Dec 7 09:20:01 2022 [120316.332954] Lustre: DEBUG MARKER: Wed Dec 7 09:25:01 2022 [120616.365969] Lustre: DEBUG MARKER: Wed Dec 7 09:30:01 2022 [120916.398774] Lustre: DEBUG MARKER: Wed Dec 7 09:35:01 2022 [121216.431105] Lustre: DEBUG MARKER: Wed Dec 7 09:40:01 2022 [121516.464241] Lustre: DEBUG MARKER: Wed Dec 7 09:45:01 2022 [121816.495789] Lustre: DEBUG MARKER: Wed Dec 7 09:50:01 2022 [122116.529465] Lustre: DEBUG MARKER: Wed Dec 7 09:55:01 2022 [122416.562110] Lustre: DEBUG MARKER: Wed Dec 7 10:00:01 2022 [122716.618475] Lustre: DEBUG MARKER: Wed Dec 7 10:05:01 2022 [123016.649897] Lustre: DEBUG MARKER: Wed Dec 7 10:10:01 2022 [123316.681809] Lustre: DEBUG MARKER: Wed Dec 7 10:15:01 2022 [123616.713036] Lustre: DEBUG MARKER: Wed Dec 7 10:20:01 2022 [123916.745273] Lustre: DEBUG MARKER: Wed Dec 7 10:25:01 2022 [124216.778364] Lustre: DEBUG MARKER: Wed Dec 7 10:30:01 2022 [124516.812437] Lustre: DEBUG MARKER: Wed Dec 7 10:35:01 2022 [124816.845573] Lustre: DEBUG MARKER: Wed Dec 7 10:40:01 2022 [125116.875844] Lustre: DEBUG MARKER: Wed Dec 7 10:45:01 2022 [125416.908013] Lustre: DEBUG MARKER: Wed Dec 7 10:50:02 2022 [125715.938169] Lustre: DEBUG MARKER: Wed Dec 7 10:55:01 2022 [126015.969244] Lustre: DEBUG MARKER: Wed Dec 7 11:00:01 2022 [126316.019708] Lustre: DEBUG MARKER: Wed Dec 7 11:05:01 2022 [126616.050561] Lustre: DEBUG MARKER: Wed Dec 7 11:10:01 2022 [126916.082903] Lustre: DEBUG MARKER: Wed Dec 7 11:15:01 2022 [127216.114744] Lustre: DEBUG MARKER: Wed Dec 7 11:20:01 2022 [127516.146790] Lustre: DEBUG MARKER: Wed Dec 7 11:25:01 2022 [127816.175287] Lustre: DEBUG MARKER: Wed Dec 7 11:30:01 2022 [128116.208467] Lustre: DEBUG MARKER: Wed Dec 7 11:35:01 2022 [128416.237746] Lustre: DEBUG MARKER: Wed Dec 7 11:40:01 2022 [128716.268085] Lustre: DEBUG MARKER: Wed Dec 7 11:45:01 2022 [129016.296490] Lustre: DEBUG MARKER: Wed Dec 7 11:50:01 2022 [129316.327292] Lustre: DEBUG MARKER: Wed Dec 7 11:55:01 2022 [129616.362414] Lustre: DEBUG MARKER: Wed Dec 7 12:00:01 2022 [129833.926440] blk_update_request: I/O error, dev fd0, sector 0 [129852.978834] nf_conntrack version 0.5.0 (65536 buckets, 262144 max) [129916.416589] Lustre: DEBUG MARKER: Wed Dec 7 12:05:01 2022 [130216.447710] Lustre: DEBUG MARKER: Wed Dec 7 12:10:01 2022 [130516.479493] Lustre: DEBUG MARKER: Wed Dec 7 12:15:01 2022 [130816.513554] Lustre: DEBUG MARKER: Wed Dec 7 12:20:01 2022 [131116.546920] Lustre: DEBUG MARKER: Wed Dec 7 12:25:01 2022 [131416.577488] Lustre: DEBUG MARKER: Wed Dec 7 12:30:01 2022 [131716.612738] Lustre: DEBUG MARKER: Wed Dec 7 12:35:01 2022 [132016.642794] Lustre: DEBUG MARKER: Wed Dec 7 12:40:01 2022 [132316.675167] Lustre: DEBUG MARKER: Wed Dec 7 12:45:01 2022 [132616.705916] Lustre: DEBUG MARKER: Wed Dec 7 12:50:01 2022 [132916.739614] Lustre: DEBUG MARKER: Wed Dec 7 12:55:01 2022 [133216.773431] Lustre: DEBUG MARKER: Wed Dec 7 13:00:01 2022 [133516.828741] Lustre: DEBUG MARKER: Wed Dec 7 13:05:01 2022 [133816.860770] Lustre: DEBUG MARKER: Wed Dec 7 13:10:01 2022 [134116.894105] Lustre: DEBUG MARKER: Wed Dec 7 13:15:01 2022 [134416.927178] Lustre: DEBUG MARKER: Wed Dec 7 13:20:01 2022 [134716.960556] Lustre: DEBUG MARKER: Wed Dec 7 13:25:01 2022 [135016.992282] Lustre: DEBUG MARKER: Wed Dec 7 13:30:01 2022 [135317.026464] Lustre: DEBUG MARKER: Wed Dec 7 13:35:01 2022 [135617.058451] Lustre: DEBUG MARKER: Wed Dec 7 13:40:01 2022 [135917.092072] Lustre: DEBUG MARKER: Wed Dec 7 13:45:01 2022 [136217.124965] Lustre: DEBUG MARKER: Wed Dec 7 13:50:01 2022 [136311.435333] blk_update_request: I/O error, dev fd0, sector 0 [136348.769934] blk_update_request: I/O error, dev fd0, sector 0 [136383.717182] blk_update_request: I/O error, dev fd0, sector 0 [136517.158064] Lustre: DEBUG MARKER: Wed Dec 7 13:55:01 2022 [136655.657959] blk_update_request: I/O error, dev fd0, sector 0 [136817.192500] Lustre: DEBUG MARKER: Wed Dec 7 14:00:01 2022 [136838.877193] blk_update_request: I/O error, dev fd0, sector 0 [136892.602376] blk_update_request: I/O error, dev fd0, sector 0 [137117.252091] Lustre: DEBUG MARKER: Wed Dec 7 14:05:01 2022 [137417.285658] Lustre: DEBUG MARKER: Wed Dec 7 14:10:01 2022 [137717.318588] Lustre: DEBUG MARKER: Wed Dec 7 14:15:01 2022 [137982.340125] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds [137982.343507] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 8 previous similar messages [137982.345187] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID19@o2ibyy (108): c: 29, oc: 0, rc: 42 [137982.347510] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 8 previous similar messages [138017.350526] Lustre: DEBUG MARKER: Wed Dec 7 14:20:01 2022 [138317.383572] Lustre: DEBUG MARKER: Wed Dec 7 14:25:01 2022 [138617.413584] Lustre: DEBUG MARKER: Wed Dec 7 14:30:01 2022 [138916.442875] Lustre: DEBUG MARKER: Wed Dec 7 14:35:01 2022 [139216.472983] Lustre: DEBUG MARKER: Wed Dec 7 14:40:01 2022 [139516.503370] Lustre: DEBUG MARKER: Wed Dec 7 14:45:01 2022 [139816.536391] Lustre: DEBUG MARKER: Wed Dec 7 14:50:01 2022 [140116.569204] Lustre: DEBUG MARKER: Wed Dec 7 14:55:01 2022 [140416.603958] Lustre: DEBUG MARKER: Wed Dec 7 15:00:01 2022 [140716.660582] Lustre: DEBUG MARKER: Wed Dec 7 15:05:01 2022 [141016.693435] Lustre: DEBUG MARKER: Wed Dec 7 15:10:01 2022 [141316.726666] Lustre: DEBUG MARKER: Wed Dec 7 15:15:01 2022 [141616.757747] Lustre: DEBUG MARKER: Wed Dec 7 15:20:01 2022 [141916.789841] Lustre: DEBUG MARKER: Wed Dec 7 15:25:01 2022 [142216.820000] Lustre: DEBUG MARKER: Wed Dec 7 15:30:01 2022 [142516.853111] Lustre: DEBUG MARKER: Wed Dec 7 15:35:01 2022 [142816.885773] Lustre: DEBUG MARKER: Wed Dec 7 15:40:01 2022 [143116.917869] Lustre: DEBUG MARKER: Wed Dec 7 15:45:01 2022 [143416.948819] Lustre: DEBUG MARKER: Wed Dec 7 15:50:01 2022 [143716.981044] Lustre: DEBUG MARKER: Wed Dec 7 15:55:01 2022 [144017.013762] Lustre: DEBUG MARKER: Wed Dec 7 16:00:01 2022 [144317.071249] Lustre: DEBUG MARKER: Wed Dec 7 16:05:01 2022 [144617.103033] Lustre: DEBUG MARKER: Wed Dec 7 16:10:01 2022 [144917.134844] Lustre: DEBUG MARKER: Wed Dec 7 16:15:01 2022 [145217.164568] Lustre: DEBUG MARKER: Wed Dec 7 16:20:01 2022 [145517.196145] Lustre: DEBUG MARKER: Wed Dec 7 16:25:01 2022 [145817.228285] Lustre: DEBUG MARKER: Wed Dec 7 16:30:01 2022 [146117.260931] Lustre: DEBUG MARKER: Wed Dec 7 16:35:01 2022 [146417.292563] Lustre: DEBUG MARKER: Wed Dec 7 16:40:01 2022 [146717.325593] Lustre: DEBUG MARKER: Wed Dec 7 16:45:01 2022 [147017.358174] Lustre: DEBUG MARKER: Wed Dec 7 16:50:01 2022 [147317.392182] Lustre: DEBUG MARKER: Wed Dec 7 16:55:01 2022 [147617.425511] Lustre: DEBUG MARKER: Wed Dec 7 17:00:01 2022 [147917.480470] Lustre: DEBUG MARKER: Wed Dec 7 17:05:01 2022 [148217.512287] Lustre: DEBUG MARKER: Wed Dec 7 17:10:01 2022 [148517.545880] Lustre: DEBUG MARKER: Wed Dec 7 17:15:01 2022 [148817.577382] Lustre: DEBUG MARKER: Wed Dec 7 17:20:01 2022 [149117.610163] Lustre: DEBUG MARKER: Wed Dec 7 17:25:01 2022 [149417.642427] Lustre: DEBUG MARKER: Wed Dec 7 17:30:01 2022 [149717.676908] Lustre: DEBUG MARKER: Wed Dec 7 17:35:01 2022 [150017.707163] Lustre: DEBUG MARKER: Wed Dec 7 17:40:01 2022 [150317.742612] Lustre: DEBUG MARKER: Wed Dec 7 17:45:01 2022 [150617.776407] Lustre: DEBUG MARKER: Wed Dec 7 17:50:01 2022 [150917.810442] Lustre: DEBUG MARKER: Wed Dec 7 17:55:01 2022 [151217.843277] Lustre: DEBUG MARKER: Wed Dec 7 18:00:01 2022 [151517.897507] Lustre: DEBUG MARKER: Wed Dec 7 18:05:01 2022 [151817.931757] Lustre: DEBUG MARKER: Wed Dec 7 18:10:02 2022 [152116.964237] Lustre: DEBUG MARKER: Wed Dec 7 18:15:01 2022 [152416.994236] Lustre: DEBUG MARKER: Wed Dec 7 18:20:01 2022 [152717.028599] Lustre: DEBUG MARKER: Wed Dec 7 18:25:01 2022 [153017.059944] Lustre: DEBUG MARKER: Wed Dec 7 18:30:01 2022 [153317.092471] Lustre: DEBUG MARKER: Wed Dec 7 18:35:01 2022 [153617.124149] Lustre: DEBUG MARKER: Wed Dec 7 18:40:01 2022 [153917.158265] Lustre: DEBUG MARKER: Wed Dec 7 18:45:01 2022 [154217.190020] Lustre: DEBUG MARKER: Wed Dec 7 18:50:01 2022 [154517.222803] Lustre: DEBUG MARKER: Wed Dec 7 18:55:01 2022 [154817.255335] Lustre: DEBUG MARKER: Wed Dec 7 19:00:01 2022 [155117.310686] Lustre: DEBUG MARKER: Wed Dec 7 19:05:01 2022 [155417.343235] Lustre: DEBUG MARKER: Wed Dec 7 19:10:01 2022 [155717.375709] Lustre: DEBUG MARKER: Wed Dec 7 19:15:01 2022 [156017.405298] Lustre: DEBUG MARKER: Wed Dec 7 19:20:01 2022 [156317.438128] Lustre: DEBUG MARKER: Wed Dec 7 19:25:01 2022 [156617.472136] Lustre: DEBUG MARKER: Wed Dec 7 19:30:01 2022 [156917.504527] Lustre: DEBUG MARKER: Wed Dec 7 19:35:01 2022 [157217.538773] Lustre: DEBUG MARKER: Wed Dec 7 19:40:01 2022 [157517.573797] Lustre: DEBUG MARKER: Wed Dec 7 19:45:01 2022 [157817.608012] Lustre: DEBUG MARKER: Wed Dec 7 19:50:01 2022 [158117.643036] Lustre: DEBUG MARKER: Wed Dec 7 19:55:01 2022 [158417.676702] Lustre: DEBUG MARKER: Wed Dec 7 20:00:01 2022 [158717.732723] Lustre: DEBUG MARKER: Wed Dec 7 20:05:01 2022 [159017.763458] Lustre: DEBUG MARKER: Wed Dec 7 20:10:01 2022 [159317.795663] Lustre: DEBUG MARKER: Wed Dec 7 20:15:01 2022 [159617.826576] Lustre: DEBUG MARKER: Wed Dec 7 20:20:01 2022 [159917.859080] Lustre: DEBUG MARKER: Wed Dec 7 20:25:01 2022 [160217.889909] Lustre: DEBUG MARKER: Wed Dec 7 20:30:01 2022 [160517.923176] Lustre: DEBUG MARKER: Wed Dec 7 20:35:01 2022 [160817.956115] Lustre: DEBUG MARKER: Wed Dec 7 20:40:01 2022 [161117.989389] Lustre: DEBUG MARKER: Wed Dec 7 20:45:01 2022 [161418.020758] Lustre: DEBUG MARKER: Wed Dec 7 20:50:01 2022 [161718.054058] Lustre: DEBUG MARKER: Wed Dec 7 20:55:01 2022 [162018.088439] Lustre: DEBUG MARKER: Wed Dec 7 21:00:01 2022 [162318.146172] Lustre: DEBUG MARKER: Wed Dec 7 21:05:01 2022 [162618.177706] Lustre: DEBUG MARKER: Wed Dec 7 21:10:01 2022 [162918.212258] Lustre: DEBUG MARKER: Wed Dec 7 21:15:01 2022 [163218.245814] Lustre: DEBUG MARKER: Wed Dec 7 21:20:01 2022 [163518.277390] Lustre: DEBUG MARKER: Wed Dec 7 21:25:01 2022 [163818.309572] Lustre: DEBUG MARKER: Wed Dec 7 21:30:01 2022 [164118.343340] Lustre: DEBUG MARKER: Wed Dec 7 21:35:01 2022 [164418.375704] Lustre: DEBUG MARKER: Wed Dec 7 21:40:01 2022 [164718.408345] Lustre: DEBUG MARKER: Wed Dec 7 21:45:01 2022 [165018.440680] Lustre: DEBUG MARKER: Wed Dec 7 21:50:02 2022 [165317.474360] Lustre: DEBUG MARKER: Wed Dec 7 21:55:01 2022 [165617.506211] Lustre: DEBUG MARKER: Wed Dec 7 22:00:01 2022 [165917.560186] Lustre: DEBUG MARKER: Wed Dec 7 22:05:01 2022 [166217.590559] Lustre: DEBUG MARKER: Wed Dec 7 22:10:01 2022 [166517.623291] Lustre: DEBUG MARKER: Wed Dec 7 22:15:01 2022 [166817.653003] Lustre: DEBUG MARKER: Wed Dec 7 22:20:01 2022 [167117.685097] Lustre: DEBUG MARKER: Wed Dec 7 22:25:01 2022 [167417.715597] Lustre: DEBUG MARKER: Wed Dec 7 22:30:01 2022 [167717.749532] Lustre: DEBUG MARKER: Wed Dec 7 22:35:01 2022 [168017.781817] Lustre: DEBUG MARKER: Wed Dec 7 22:40:01 2022 [168317.815479] Lustre: DEBUG MARKER: Wed Dec 7 22:45:01 2022 [168617.846678] Lustre: DEBUG MARKER: Wed Dec 7 22:50:01 2022 [168917.878377] Lustre: DEBUG MARKER: Wed Dec 7 22:55:01 2022 [169217.911720] Lustre: DEBUG MARKER: Wed Dec 7 23:00:01 2022 [169517.966483] Lustre: DEBUG MARKER: Wed Dec 7 23:05:01 2022 [169817.998959] Lustre: DEBUG MARKER: Wed Dec 7 23:10:01 2022 [170118.032575] Lustre: DEBUG MARKER: Wed Dec 7 23:15:01 2022 [170418.063841] Lustre: DEBUG MARKER: Wed Dec 7 23:20:01 2022 [170718.098091] Lustre: DEBUG MARKER: Wed Dec 7 23:25:01 2022 [171018.128605] Lustre: DEBUG MARKER: Wed Dec 7 23:30:01 2022 [171318.161348] Lustre: DEBUG MARKER: Wed Dec 7 23:35:01 2022 [171618.192626] Lustre: DEBUG MARKER: Wed Dec 7 23:40:01 2022 [171918.225708] Lustre: DEBUG MARKER: Wed Dec 7 23:45:01 2022 [172218.256715] Lustre: DEBUG MARKER: Wed Dec 7 23:50:01 2022 [172518.461156] Lustre: DEBUG MARKER: Wed Dec 7 23:55:01 2022 [172818.494141] Lustre: DEBUG MARKER: Thu Dec 8 00:00:01 2022 [173118.550424] Lustre: DEBUG MARKER: Thu Dec 8 00:05:01 2022 [173418.581068] Lustre: DEBUG MARKER: Thu Dec 8 00:10:01 2022 [173718.614754] Lustre: DEBUG MARKER: Thu Dec 8 00:15:01 2022 [174018.648268] Lustre: DEBUG MARKER: Thu Dec 8 00:20:01 2022 [174318.681567] Lustre: DEBUG MARKER: Thu Dec 8 00:25:01 2022 [174618.714889] Lustre: DEBUG MARKER: Thu Dec 8 00:30:01 2022 [174918.748030] Lustre: DEBUG MARKER: Thu Dec 8 00:35:01 2022 [175218.779560] Lustre: DEBUG MARKER: Thu Dec 8 00:40:01 2022 [175518.811426] Lustre: DEBUG MARKER: Thu Dec 8 00:45:01 2022 [175818.843371] Lustre: DEBUG MARKER: Thu Dec 8 00:50:01 2022 [176118.876414] Lustre: DEBUG MARKER: Thu Dec 8 00:55:02 2022 [176417.910617] Lustre: DEBUG MARKER: Thu Dec 8 01:00:01 2022 [176717.969673] Lustre: DEBUG MARKER: Thu Dec 8 01:05:01 2022 [177018.001620] Lustre: DEBUG MARKER: Thu Dec 8 01:10:01 2022 [177318.034003] Lustre: DEBUG MARKER: Thu Dec 8 01:15:01 2022 [177618.064717] Lustre: DEBUG MARKER: Thu Dec 8 01:20:01 2022 [177918.100918] Lustre: DEBUG MARKER: Thu Dec 8 01:25:01 2022 [178218.132574] Lustre: DEBUG MARKER: Thu Dec 8 01:30:01 2022 [178518.164780] Lustre: DEBUG MARKER: Thu Dec 8 01:35:01 2022 [178818.195794] Lustre: DEBUG MARKER: Thu Dec 8 01:40:01 2022 [179118.227961] Lustre: DEBUG MARKER: Thu Dec 8 01:45:01 2022 [179418.259037] Lustre: DEBUG MARKER: Thu Dec 8 01:50:01 2022 [179718.293139] Lustre: DEBUG MARKER: Thu Dec 8 01:55:01 2022 [180018.328962] Lustre: DEBUG MARKER: Thu Dec 8 02:00:01 2022 [180318.383592] Lustre: DEBUG MARKER: Thu Dec 8 02:05:01 2022 [180618.411633] Lustre: DEBUG MARKER: Thu Dec 8 02:10:01 2022 [180918.442381] Lustre: DEBUG MARKER: Thu Dec 8 02:15:01 2022 [181218.475126] Lustre: DEBUG MARKER: Thu Dec 8 02:20:01 2022 [181518.504894] Lustre: DEBUG MARKER: Thu Dec 8 02:25:01 2022 [181818.533660] Lustre: DEBUG MARKER: Thu Dec 8 02:30:01 2022 [182118.565281] Lustre: DEBUG MARKER: Thu Dec 8 02:35:01 2022 [182418.597661] Lustre: DEBUG MARKER: Thu Dec 8 02:40:01 2022 [182718.630543] Lustre: DEBUG MARKER: Thu Dec 8 02:45:01 2022 [183018.658771] Lustre: DEBUG MARKER: Thu Dec 8 02:50:01 2022 [183318.691361] Lustre: DEBUG MARKER: Thu Dec 8 02:55:01 2022 [183618.725648] Lustre: DEBUG MARKER: Thu Dec 8 03:00:01 2022 [183918.781207] Lustre: DEBUG MARKER: Thu Dec 8 03:05:01 2022 [184218.811515] Lustre: DEBUG MARKER: Thu Dec 8 03:10:01 2022 [184518.843235] Lustre: DEBUG MARKER: Thu Dec 8 03:15:01 2022 [184818.873156] Lustre: DEBUG MARKER: Thu Dec 8 03:20:01 2022 [185118.904493] Lustre: DEBUG MARKER: Thu Dec 8 03:25:01 2022 [185418.935253] Lustre: DEBUG MARKER: Thu Dec 8 03:30:01 2022 [185718.967413] Lustre: DEBUG MARKER: Thu Dec 8 03:35:01 2022 [186018.997631] Lustre: DEBUG MARKER: Thu Dec 8 03:40:01 2022 [186319.030845] Lustre: DEBUG MARKER: Thu Dec 8 03:45:01 2022 [186619.061498] Lustre: DEBUG MARKER: Thu Dec 8 03:50:01 2022 [186919.095083] Lustre: DEBUG MARKER: Thu Dec 8 03:55:01 2022 [187219.130706] Lustre: DEBUG MARKER: Thu Dec 8 04:00:01 2022 [187519.189193] Lustre: DEBUG MARKER: Thu Dec 8 04:05:01 2022 [187819.219985] Lustre: DEBUG MARKER: Thu Dec 8 04:10:01 2022 [188119.253792] Lustre: DEBUG MARKER: Thu Dec 8 04:15:01 2022 [188419.285087] Lustre: DEBUG MARKER: Thu Dec 8 04:20:01 2022 [188719.317749] Lustre: DEBUG MARKER: Thu Dec 8 04:25:01 2022 [189019.349057] Lustre: DEBUG MARKER: Thu Dec 8 04:30:01 2022 [189319.382631] Lustre: DEBUG MARKER: Thu Dec 8 04:35:01 2022 [189618.415562] Lustre: DEBUG MARKER: Thu Dec 8 04:40:01 2022 [189918.447596] Lustre: DEBUG MARKER: Thu Dec 8 04:45:01 2022 [190218.478180] Lustre: DEBUG MARKER: Thu Dec 8 04:50:01 2022 [190518.510587] Lustre: DEBUG MARKER: Thu Dec 8 04:55:01 2022 [190818.544642] Lustre: DEBUG MARKER: Thu Dec 8 05:00:01 2022 [191118.601651] Lustre: DEBUG MARKER: Thu Dec 8 05:05:01 2022 [191418.633622] Lustre: DEBUG MARKER: Thu Dec 8 05:10:01 2022 [191718.666841] Lustre: DEBUG MARKER: Thu Dec 8 05:15:01 2022 [192018.700646] Lustre: DEBUG MARKER: Thu Dec 8 05:20:01 2022 [192318.732934] Lustre: DEBUG MARKER: Thu Dec 8 05:25:01 2022 [192618.764379] Lustre: DEBUG MARKER: Thu Dec 8 05:30:01 2022 [192918.796302] Lustre: DEBUG MARKER: Thu Dec 8 05:35:01 2022 [193218.828677] Lustre: DEBUG MARKER: Thu Dec 8 05:40:01 2022 [193518.861559] Lustre: DEBUG MARKER: Thu Dec 8 05:45:01 2022 [193818.892034] Lustre: DEBUG MARKER: Thu Dec 8 05:50:01 2022 [194118.925136] Lustre: DEBUG MARKER: Thu Dec 8 05:55:01 2022 [194418.960094] Lustre: DEBUG MARKER: Thu Dec 8 06:00:01 2022 [194719.016099] Lustre: DEBUG MARKER: Thu Dec 8 06:05:01 2022 [195019.048113] Lustre: DEBUG MARKER: Thu Dec 8 06:10:01 2022 [195319.080965] Lustre: DEBUG MARKER: Thu Dec 8 06:15:01 2022 [195619.112055] Lustre: DEBUG MARKER: Thu Dec 8 06:20:01 2022 [195919.145346] Lustre: DEBUG MARKER: Thu Dec 8 06:25:01 2022 [196219.176840] Lustre: DEBUG MARKER: Thu Dec 8 06:30:01 2022 [196519.209592] Lustre: DEBUG MARKER: Thu Dec 8 06:35:01 2022 [196819.240367] Lustre: DEBUG MARKER: Thu Dec 8 06:40:01 2022 [197119.275676] Lustre: DEBUG MARKER: Thu Dec 8 06:45:01 2022 [197419.306254] Lustre: DEBUG MARKER: Thu Dec 8 06:50:01 2022 [197719.339557] Lustre: DEBUG MARKER: Thu Dec 8 06:55:01 2022 [198019.373411] Lustre: DEBUG MARKER: Thu Dec 8 07:00:01 2022 [198319.428284] Lustre: DEBUG MARKER: Thu Dec 8 07:05:01 2022 [198619.460328] Lustre: DEBUG MARKER: Thu Dec 8 07:10:01 2022 [198919.494213] Lustre: DEBUG MARKER: Thu Dec 8 07:15:01 2022 [199219.527026] Lustre: DEBUG MARKER: Thu Dec 8 07:20:01 2022 [199519.558590] Lustre: DEBUG MARKER: Thu Dec 8 07:25:01 2022 [199819.589695] Lustre: DEBUG MARKER: Thu Dec 8 07:30:01 2022 [200119.622757] Lustre: DEBUG MARKER: Thu Dec 8 07:35:01 2022 [200419.655604] Lustre: DEBUG MARKER: Thu Dec 8 07:40:01 2022 [200719.689289] Lustre: DEBUG MARKER: Thu Dec 8 07:45:01 2022 [201019.720048] Lustre: DEBUG MARKER: Thu Dec 8 07:50:01 2022 [201319.753550] Lustre: DEBUG MARKER: Thu Dec 8 07:55:01 2022 [201619.788299] Lustre: DEBUG MARKER: Thu Dec 8 08:00:01 2022 [201919.844540] Lustre: DEBUG MARKER: Thu Dec 8 08:05:01 2022 [202219.877147] Lustre: DEBUG MARKER: Thu Dec 8 08:10:01 2022 [202518.910658] Lustre: DEBUG MARKER: Thu Dec 8 08:15:01 2022 [202818.942193] Lustre: DEBUG MARKER: Thu Dec 8 08:20:01 2022 [203118.974506] Lustre: DEBUG MARKER: Thu Dec 8 08:25:01 2022 [203419.006322] Lustre: DEBUG MARKER: Thu Dec 8 08:30:01 2022 [203719.038702] Lustre: DEBUG MARKER: Thu Dec 8 08:35:01 2022 [204019.069635] Lustre: DEBUG MARKER: Thu Dec 8 08:40:01 2022 [204319.101753] Lustre: DEBUG MARKER: Thu Dec 8 08:45:01 2022 [204619.135076] Lustre: DEBUG MARKER: Thu Dec 8 08:50:01 2022 [204919.167299] Lustre: DEBUG MARKER: Thu Dec 8 08:55:01 2022 [205219.200831] Lustre: DEBUG MARKER: Thu Dec 8 09:00:01 2022 [205519.256837] Lustre: DEBUG MARKER: Thu Dec 8 09:05:01 2022 [205819.287637] Lustre: DEBUG MARKER: Thu Dec 8 09:10:01 2022 [206119.320414] Lustre: DEBUG MARKER: Thu Dec 8 09:15:01 2022 [206419.352057] Lustre: DEBUG MARKER: Thu Dec 8 09:20:01 2022 [206719.385527] Lustre: DEBUG MARKER: Thu Dec 8 09:25:01 2022 [207019.416583] Lustre: DEBUG MARKER: Thu Dec 8 09:30:01 2022 [207319.449475] Lustre: DEBUG MARKER: Thu Dec 8 09:35:01 2022 [207619.480692] Lustre: DEBUG MARKER: Thu Dec 8 09:40:01 2022 [207919.512761] Lustre: DEBUG MARKER: Thu Dec 8 09:45:01 2022 [208219.544476] Lustre: DEBUG MARKER: Thu Dec 8 09:50:01 2022 [208519.578369] Lustre: DEBUG MARKER: Thu Dec 8 09:55:01 2022 [208819.611586] Lustre: DEBUG MARKER: Thu Dec 8 10:00:01 2022 [209119.669291] Lustre: DEBUG MARKER: Thu Dec 8 10:05:01 2022 [209419.702131] Lustre: DEBUG MARKER: Thu Dec 8 10:10:01 2022 [209719.732026] Lustre: DEBUG MARKER: Thu Dec 8 10:15:01 2022 [210019.760264] Lustre: DEBUG MARKER: Thu Dec 8 10:20:01 2022 [210319.791711] Lustre: DEBUG MARKER: Thu Dec 8 10:25:01 2022 [210619.821826] Lustre: DEBUG MARKER: Thu Dec 8 10:30:01 2022 [210919.852660] Lustre: DEBUG MARKER: Thu Dec 8 10:35:01 2022 [211219.882466] Lustre: DEBUG MARKER: Thu Dec 8 10:40:01 2022 [211519.914079] Lustre: DEBUG MARKER: Thu Dec 8 10:45:01 2022 [211819.942642] Lustre: DEBUG MARKER: Thu Dec 8 10:50:01 2022 [212119.973725] Lustre: DEBUG MARKER: Thu Dec 8 10:55:01 2022 [212420.004571] Lustre: DEBUG MARKER: Thu Dec 8 11:00:01 2022 [212720.056579] Lustre: DEBUG MARKER: Thu Dec 8 11:05:01 2022 [213020.085862] Lustre: DEBUG MARKER: Thu Dec 8 11:10:01 2022 [213320.117192] Lustre: DEBUG MARKER: Thu Dec 8 11:15:01 2022 [213620.145288] Lustre: DEBUG MARKER: Thu Dec 8 11:20:01 2022 [213920.178049] Lustre: DEBUG MARKER: Thu Dec 8 11:25:01 2022 [214220.207379] Lustre: DEBUG MARKER: Thu Dec 8 11:30:01 2022 [214520.239071] Lustre: DEBUG MARKER: Thu Dec 8 11:35:01 2022 [214820.269493] Lustre: DEBUG MARKER: Thu Dec 8 11:40:01 2022 [215120.300221] Lustre: DEBUG MARKER: Thu Dec 8 11:45:01 2022 [215420.330207] Lustre: DEBUG MARKER: Thu Dec 8 11:50:01 2022 [215720.360969] Lustre: DEBUG MARKER: Thu Dec 8 11:55:01 2022 [216020.391477] Lustre: DEBUG MARKER: Thu Dec 8 12:00:01 2022 [216319.447968] Lustre: DEBUG MARKER: Thu Dec 8 12:05:01 2022 [216619.477718] Lustre: DEBUG MARKER: Thu Dec 8 12:10:01 2022 [216919.508639] Lustre: DEBUG MARKER: Thu Dec 8 12:15:01 2022 [217219.537466] Lustre: DEBUG MARKER: Thu Dec 8 12:20:01 2022 [217519.567739] Lustre: DEBUG MARKER: Thu Dec 8 12:25:01 2022 [217819.597953] Lustre: DEBUG MARKER: Thu Dec 8 12:30:01 2022 [218119.629038] Lustre: DEBUG MARKER: Thu Dec 8 12:35:01 2022 [218419.659780] Lustre: DEBUG MARKER: Thu Dec 8 12:40:01 2022 [218719.691388] Lustre: DEBUG MARKER: Thu Dec 8 12:45:01 2022 [219019.720459] Lustre: DEBUG MARKER: Thu Dec 8 12:50:01 2022 [219319.751914] Lustre: DEBUG MARKER: Thu Dec 8 12:55:01 2022 [219619.781291] Lustre: DEBUG MARKER: Thu Dec 8 13:00:01 2022 [219919.833826] Lustre: DEBUG MARKER: Thu Dec 8 13:05:01 2022 [220219.862216] Lustre: DEBUG MARKER: Thu Dec 8 13:10:01 2022 [220519.892683] Lustre: DEBUG MARKER: Thu Dec 8 13:15:01 2022 [220819.921399] Lustre: DEBUG MARKER: Thu Dec 8 13:20:01 2022 [221119.952514] Lustre: DEBUG MARKER: Thu Dec 8 13:25:01 2022 [221419.983156] Lustre: DEBUG MARKER: Thu Dec 8 13:30:01 2022 [221720.014532] Lustre: DEBUG MARKER: Thu Dec 8 13:35:01 2022 [222020.044662] Lustre: DEBUG MARKER: Thu Dec 8 13:40:01 2022 [222320.073988] Lustre: DEBUG MARKER: Thu Dec 8 13:45:01 2022 [222620.103187] Lustre: DEBUG MARKER: Thu Dec 8 13:50:01 2022 [222920.133275] Lustre: DEBUG MARKER: Thu Dec 8 13:55:01 2022 [223220.165874] Lustre: DEBUG MARKER: Thu Dec 8 14:00:01 2022 [223520.216725] Lustre: DEBUG MARKER: Thu Dec 8 14:05:01 2022 [223820.247150] Lustre: DEBUG MARKER: Thu Dec 8 14:10:01 2022 [224120.278159] Lustre: DEBUG MARKER: Thu Dec 8 14:15:01 2022 [224420.307474] Lustre: DEBUG MARKER: Thu Dec 8 14:20:01 2022 [224720.338289] Lustre: DEBUG MARKER: Thu Dec 8 14:25:01 2022 [225020.367455] Lustre: DEBUG MARKER: Thu Dec 8 14:30:01 2022 [225320.398448] Lustre: DEBUG MARKER: Thu Dec 8 14:35:01 2022 [225620.428826] Lustre: DEBUG MARKER: Thu Dec 8 14:40:01 2022 [225920.459997] Lustre: DEBUG MARKER: Thu Dec 8 14:45:01 2022 [226220.492362] Lustre: DEBUG MARKER: Thu Dec 8 14:50:01 2022 [226520.523669] Lustre: DEBUG MARKER: Thu Dec 8 14:55:01 2022 [226820.557895] Lustre: DEBUG MARKER: Thu Dec 8 15:00:01 2022 [227120.609246] Lustre: DEBUG MARKER: Thu Dec 8 15:05:01 2022 [227420.638691] Lustre: DEBUG MARKER: Thu Dec 8 15:10:01 2022 [227720.671893] Lustre: DEBUG MARKER: Thu Dec 8 15:15:01 2022 [228020.703323] Lustre: DEBUG MARKER: Thu Dec 8 15:20:01 2022 [228320.734694] Lustre: DEBUG MARKER: Thu Dec 8 15:25:01 2022 [228620.762574] Lustre: DEBUG MARKER: Thu Dec 8 15:30:01 2022 [228920.794894] Lustre: DEBUG MARKER: Thu Dec 8 15:35:01 2022 [229220.823057] Lustre: DEBUG MARKER: Thu Dec 8 15:40:01 2022 [229520.857033] Lustre: DEBUG MARKER: Thu Dec 8 15:45:01 2022 [229820.888844] Lustre: DEBUG MARKER: Thu Dec 8 15:50:01 2022 [230120.921225] Lustre: DEBUG MARKER: Thu Dec 8 15:55:01 2022 [230420.952892] Lustre: DEBUG MARKER: Thu Dec 8 16:00:01 2022 [230720.007499] Lustre: DEBUG MARKER: Thu Dec 8 16:05:01 2022 [231020.036042] Lustre: DEBUG MARKER: Thu Dec 8 16:10:01 2022 [231320.067837] Lustre: DEBUG MARKER: Thu Dec 8 16:15:01 2022 [231620.096362] Lustre: DEBUG MARKER: Thu Dec 8 16:20:01 2022 [231920.127441] Lustre: DEBUG MARKER: Thu Dec 8 16:25:01 2022 [232220.158338] Lustre: DEBUG MARKER: Thu Dec 8 16:30:01 2022 [232520.188699] Lustre: DEBUG MARKER: Thu Dec 8 16:35:01 2022 [232820.218433] Lustre: DEBUG MARKER: Thu Dec 8 16:40:01 2022 [233120.248710] Lustre: DEBUG MARKER: Thu Dec 8 16:45:01 2022 [233420.277686] Lustre: DEBUG MARKER: Thu Dec 8 16:50:01 2022 [233720.307894] Lustre: DEBUG MARKER: Thu Dec 8 16:55:01 2022 [234020.339929] Lustre: DEBUG MARKER: Thu Dec 8 17:00:01 2022 [234320.391687] Lustre: DEBUG MARKER: Thu Dec 8 17:05:01 2022 [234620.420762] Lustre: DEBUG MARKER: Thu Dec 8 17:10:01 2022 [234920.450871] Lustre: DEBUG MARKER: Thu Dec 8 17:15:01 2022 [235220.479140] Lustre: DEBUG MARKER: Thu Dec 8 17:20:01 2022 [235520.511683] Lustre: DEBUG MARKER: Thu Dec 8 17:25:01 2022 [235820.542339] Lustre: DEBUG MARKER: Thu Dec 8 17:30:01 2022 [236120.572096] Lustre: DEBUG MARKER: Thu Dec 8 17:35:01 2022 [236420.601529] Lustre: DEBUG MARKER: Thu Dec 8 17:40:01 2022 [236720.633203] Lustre: DEBUG MARKER: Thu Dec 8 17:45:01 2022 [237020.665157] Lustre: DEBUG MARKER: Thu Dec 8 17:50:01 2022 [237320.695447] Lustre: DEBUG MARKER: Thu Dec 8 17:55:01 2022 [237620.727161] Lustre: DEBUG MARKER: Thu Dec 8 18:00:01 2022 [237803.788787] LNet: 30630:0:(o2iblnd_cb.c:2565:kiblnd_passive_connect()) Can't accept conn from AA.AA.ID20@o2ibxx (version 12): max_frags 257 too large (256 wanted) [237803.793041] LNet: 30630:0:(o2iblnd_cb.c:2565:kiblnd_passive_connect()) Skipped 22 previous similar messages [237920.781000] Lustre: DEBUG MARKER: Thu Dec 8 18:05:01 2022 [238220.810327] Lustre: DEBUG MARKER: Thu Dec 8 18:10:01 2022 [238520.841505] Lustre: DEBUG MARKER: Thu Dec 8 18:15:01 2022 [238820.874450] Lustre: DEBUG MARKER: Thu Dec 8 18:20:01 2022 [239120.905971] Lustre: DEBUG MARKER: Thu Dec 8 18:25:01 2022 [239420.936540] Lustre: DEBUG MARKER: Thu Dec 8 18:30:01 2022 [239720.971329] Lustre: DEBUG MARKER: Thu Dec 8 18:35:01 2022 [240021.005936] Lustre: DEBUG MARKER: Thu Dec 8 18:40:01 2022 [240321.040202] Lustre: DEBUG MARKER: Thu Dec 8 18:45:01 2022 [240621.071231] Lustre: DEBUG MARKER: Thu Dec 8 18:50:01 2022 [240921.105587] Lustre: DEBUG MARKER: Thu Dec 8 18:55:01 2022 [241221.140610] Lustre: DEBUG MARKER: Thu Dec 8 19:00:01 2022 [241521.199718] Lustre: DEBUG MARKER: Thu Dec 8 19:05:01 2022 [241821.232656] Lustre: DEBUG MARKER: Thu Dec 8 19:10:01 2022 [242121.266932] Lustre: DEBUG MARKER: Thu Dec 8 19:15:01 2022 [242421.298289] Lustre: DEBUG MARKER: Thu Dec 8 19:20:01 2022 [242721.331350] Lustre: DEBUG MARKER: Thu Dec 8 19:25:01 2022 [243021.365895] Lustre: DEBUG MARKER: Thu Dec 8 19:30:01 2022 [243321.398800] Lustre: DEBUG MARKER: Thu Dec 8 19:35:01 2022 [243621.432329] Lustre: DEBUG MARKER: Thu Dec 8 19:40:01 2022 [243921.465655] Lustre: DEBUG MARKER: Thu Dec 8 19:45:01 2022 [244221.496349] Lustre: DEBUG MARKER: Thu Dec 8 19:50:01 2022 [244521.530216] Lustre: DEBUG MARKER: Thu Dec 8 19:55:02 2022 [244820.563404] Lustre: DEBUG MARKER: Thu Dec 8 20:00:01 2022 [245120.619536] Lustre: DEBUG MARKER: Thu Dec 8 20:05:01 2022 [245420.652025] Lustre: DEBUG MARKER: Thu Dec 8 20:10:01 2022 [245720.685537] Lustre: DEBUG MARKER: Thu Dec 8 20:15:01 2022 [246020.718358] Lustre: DEBUG MARKER: Thu Dec 8 20:20:01 2022 [246320.749787] Lustre: DEBUG MARKER: Thu Dec 8 20:25:01 2022 [246620.781406] Lustre: DEBUG MARKER: Thu Dec 8 20:30:01 2022 [246920.814954] Lustre: DEBUG MARKER: Thu Dec 8 20:35:01 2022 [247220.847551] Lustre: DEBUG MARKER: Thu Dec 8 20:40:01 2022 [247520.880114] Lustre: DEBUG MARKER: Thu Dec 8 20:45:01 2022 [247820.914224] Lustre: DEBUG MARKER: Thu Dec 8 20:50:01 2022 [248120.948471] Lustre: DEBUG MARKER: Thu Dec 8 20:55:01 2022 [248420.982356] Lustre: DEBUG MARKER: Thu Dec 8 21:00:01 2022 [248721.038759] Lustre: DEBUG MARKER: Thu Dec 8 21:05:01 2022 [249021.069935] Lustre: DEBUG MARKER: Thu Dec 8 21:10:01 2022 [249321.102772] Lustre: DEBUG MARKER: Thu Dec 8 21:15:01 2022 [249621.134581] Lustre: DEBUG MARKER: Thu Dec 8 21:20:01 2022 [249921.167053] Lustre: DEBUG MARKER: Thu Dec 8 21:25:01 2022 [250221.198791] Lustre: DEBUG MARKER: Thu Dec 8 21:30:01 2022 [250521.233125] Lustre: DEBUG MARKER: Thu Dec 8 21:35:01 2022 [250821.265042] Lustre: DEBUG MARKER: Thu Dec 8 21:40:01 2022 [251121.298483] Lustre: DEBUG MARKER: Thu Dec 8 21:45:01 2022 [251421.330306] Lustre: DEBUG MARKER: Thu Dec 8 21:50:01 2022 [251721.362977] Lustre: DEBUG MARKER: Thu Dec 8 21:55:01 2022 [252021.395600] Lustre: DEBUG MARKER: Thu Dec 8 22:00:01 2022 [252321.450440] Lustre: DEBUG MARKER: Thu Dec 8 22:05:01 2022 [252621.481703] Lustre: DEBUG MARKER: Thu Dec 8 22:10:01 2022 [252921.515816] Lustre: DEBUG MARKER: Thu Dec 8 22:15:01 2022 [253221.548054] Lustre: DEBUG MARKER: Thu Dec 8 22:20:01 2022 [253521.581977] Lustre: DEBUG MARKER: Thu Dec 8 22:25:01 2022 [253821.613362] Lustre: DEBUG MARKER: Thu Dec 8 22:30:01 2022 [254121.647514] Lustre: DEBUG MARKER: Thu Dec 8 22:35:01 2022 [254421.679877] Lustre: DEBUG MARKER: Thu Dec 8 22:40:01 2022 [254721.712546] Lustre: DEBUG MARKER: Thu Dec 8 22:45:01 2022 [255021.744083] Lustre: DEBUG MARKER: Thu Dec 8 22:50:01 2022 [255321.777359] Lustre: DEBUG MARKER: Thu Dec 8 22:55:01 2022 [255621.809411] Lustre: DEBUG MARKER: Thu Dec 8 23:00:01 2022 [255921.867431] Lustre: DEBUG MARKER: Thu Dec 8 23:05:01 2022 [256221.899211] Lustre: DEBUG MARKER: Thu Dec 8 23:10:01 2022 [256521.931770] Lustre: DEBUG MARKER: Thu Dec 8 23:15:01 2022 [256821.963451] Lustre: DEBUG MARKER: Thu Dec 8 23:20:01 2022 [257121.997782] Lustre: DEBUG MARKER: Thu Dec 8 23:25:01 2022 [257422.029144] Lustre: DEBUG MARKER: Thu Dec 8 23:30:02 2022 [257721.061831] Lustre: DEBUG MARKER: Thu Dec 8 23:35:01 2022 [258021.095190] Lustre: DEBUG MARKER: Thu Dec 8 23:40:01 2022 [258321.127381] Lustre: DEBUG MARKER: Thu Dec 8 23:45:01 2022 [258621.159926] Lustre: DEBUG MARKER: Thu Dec 8 23:50:01 2022 [258921.302872] Lustre: DEBUG MARKER: Thu Dec 8 23:55:01 2022 [259221.335492] Lustre: DEBUG MARKER: Fri Dec 9 00:00:01 2022 [259521.392515] Lustre: DEBUG MARKER: Fri Dec 9 00:05:01 2022 [259821.423991] Lustre: DEBUG MARKER: Fri Dec 9 00:10:01 2022 [260121.458543] Lustre: DEBUG MARKER: Fri Dec 9 00:15:01 2022 [260421.489009] Lustre: DEBUG MARKER: Fri Dec 9 00:20:01 2022 [260721.523083] Lustre: DEBUG MARKER: Fri Dec 9 00:25:01 2022 [261021.555094] Lustre: DEBUG MARKER: Fri Dec 9 00:30:01 2022 [261321.587337] Lustre: DEBUG MARKER: Fri Dec 9 00:35:01 2022 [261621.618496] Lustre: DEBUG MARKER: Fri Dec 9 00:40:01 2022 [261921.651821] Lustre: DEBUG MARKER: Fri Dec 9 00:45:01 2022 [262221.683734] Lustre: DEBUG MARKER: Fri Dec 9 00:50:01 2022 [262521.717033] Lustre: DEBUG MARKER: Fri Dec 9 00:55:01 2022 [262821.752132] Lustre: DEBUG MARKER: Fri Dec 9 01:00:01 2022 [263121.808978] Lustre: DEBUG MARKER: Fri Dec 9 01:05:01 2022 [263421.841476] Lustre: DEBUG MARKER: Fri Dec 9 01:10:01 2022 [263721.873469] Lustre: DEBUG MARKER: Fri Dec 9 01:15:01 2022 [264021.905268] Lustre: DEBUG MARKER: Fri Dec 9 01:20:01 2022 [264321.938531] Lustre: DEBUG MARKER: Fri Dec 9 01:25:01 2022 [264621.970282] Lustre: DEBUG MARKER: Fri Dec 9 01:30:01 2022 [264922.004852] Lustre: DEBUG MARKER: Fri Dec 9 01:35:01 2022 [265222.037668] Lustre: DEBUG MARKER: Fri Dec 9 01:40:01 2022 [265522.071601] Lustre: DEBUG MARKER: Fri Dec 9 01:45:01 2022 [265822.105246] Lustre: DEBUG MARKER: Fri Dec 9 01:50:01 2022 [266122.138248] Lustre: DEBUG MARKER: Fri Dec 9 01:55:01 2022 [266422.171684] Lustre: DEBUG MARKER: Fri Dec 9 02:00:01 2022 [266722.230989] Lustre: DEBUG MARKER: Fri Dec 9 02:05:01 2022 [267022.262136] Lustre: DEBUG MARKER: Fri Dec 9 02:10:01 2022 [267322.295447] Lustre: DEBUG MARKER: Fri Dec 9 02:15:01 2022 [267622.328864] Lustre: DEBUG MARKER: Fri Dec 9 02:20:01 2022 [267922.364596] Lustre: DEBUG MARKER: Fri Dec 9 02:25:01 2022 [268222.397464] Lustre: DEBUG MARKER: Fri Dec 9 02:30:01 2022 [268522.429346] Lustre: DEBUG MARKER: Fri Dec 9 02:35:01 2022 [268822.461899] Lustre: DEBUG MARKER: Fri Dec 9 02:40:01 2022 [269121.494496] Lustre: DEBUG MARKER: Fri Dec 9 02:45:01 2022 [269421.526612] Lustre: DEBUG MARKER: Fri Dec 9 02:50:01 2022 [269721.560139] Lustre: DEBUG MARKER: Fri Dec 9 02:55:01 2022 [270021.595154] Lustre: DEBUG MARKER: Fri Dec 9 03:00:01 2022 [270321.653282] Lustre: DEBUG MARKER: Fri Dec 9 03:05:01 2022 [270621.684721] Lustre: DEBUG MARKER: Fri Dec 9 03:10:01 2022 [270921.715632] Lustre: DEBUG MARKER: Fri Dec 9 03:15:01 2022 [271221.744660] Lustre: DEBUG MARKER: Fri Dec 9 03:20:01 2022 [271521.774599] Lustre: DEBUG MARKER: Fri Dec 9 03:25:01 2022 [271821.803833] Lustre: DEBUG MARKER: Fri Dec 9 03:30:01 2022 [272121.834569] Lustre: DEBUG MARKER: Fri Dec 9 03:35:01 2022 [272421.867145] Lustre: DEBUG MARKER: Fri Dec 9 03:40:01 2022 [272721.900777] Lustre: DEBUG MARKER: Fri Dec 9 03:45:01 2022 [273021.935323] Lustre: DEBUG MARKER: Fri Dec 9 03:50:01 2022 [273321.969199] Lustre: DEBUG MARKER: Fri Dec 9 03:55:01 2022 [273622.005413] Lustre: DEBUG MARKER: Fri Dec 9 04:00:01 2022 [273922.061299] Lustre: DEBUG MARKER: Fri Dec 9 04:05:01 2022 [274222.093262] Lustre: DEBUG MARKER: Fri Dec 9 04:10:01 2022 [274522.128074] Lustre: DEBUG MARKER: Fri Dec 9 04:15:01 2022 [274822.158646] Lustre: DEBUG MARKER: Fri Dec 9 04:20:01 2022 [275122.192185] Lustre: DEBUG MARKER: Fri Dec 9 04:25:01 2022 [275422.222544] Lustre: DEBUG MARKER: Fri Dec 9 04:30:01 2022 [275722.255170] Lustre: DEBUG MARKER: Fri Dec 9 04:35:01 2022 [276022.288762] Lustre: DEBUG MARKER: Fri Dec 9 04:40:01 2022 [276322.321461] Lustre: DEBUG MARKER: Fri Dec 9 04:45:01 2022 [276622.352558] Lustre: DEBUG MARKER: Fri Dec 9 04:50:01 2022 [276922.384643] Lustre: DEBUG MARKER: Fri Dec 9 04:55:01 2022 [277222.419961] Lustre: DEBUG MARKER: Fri Dec 9 05:00:01 2022 [277522.473213] Lustre: DEBUG MARKER: Fri Dec 9 05:05:01 2022 [277822.504845] Lustre: DEBUG MARKER: Fri Dec 9 05:10:01 2022 [278122.538725] Lustre: DEBUG MARKER: Fri Dec 9 05:15:01 2022 [278422.572325] Lustre: DEBUG MARKER: Fri Dec 9 05:20:01 2022 [278722.605379] Lustre: DEBUG MARKER: Fri Dec 9 05:25:01 2022 [279022.636941] Lustre: DEBUG MARKER: Fri Dec 9 05:30:01 2022 [279322.671674] Lustre: DEBUG MARKER: Fri Dec 9 05:35:01 2022 [279622.705330] Lustre: DEBUG MARKER: Fri Dec 9 05:40:01 2022 [279922.738862] Lustre: DEBUG MARKER: Fri Dec 9 05:45:01 2022 [280222.770308] Lustre: DEBUG MARKER: Fri Dec 9 05:50:01 2022 [280522.804456] Lustre: DEBUG MARKER: Fri Dec 9 05:55:01 2022 [280822.839641] Lustre: DEBUG MARKER: Fri Dec 9 06:00:01 2022 [281122.895177] Lustre: DEBUG MARKER: Fri Dec 9 06:05:01 2022 [281422.927757] Lustre: DEBUG MARKER: Fri Dec 9 06:10:01 2022 [281722.959915] Lustre: DEBUG MARKER: Fri Dec 9 06:15:01 2022 [282021.990502] Lustre: DEBUG MARKER: Fri Dec 9 06:20:01 2022 [282322.023513] Lustre: DEBUG MARKER: Fri Dec 9 06:25:01 2022 [282622.055503] Lustre: DEBUG MARKER: Fri Dec 9 06:30:01 2022 [282922.088092] Lustre: DEBUG MARKER: Fri Dec 9 06:35:01 2022 [283222.119371] Lustre: DEBUG MARKER: Fri Dec 9 06:40:01 2022 [283522.150692] Lustre: DEBUG MARKER: Fri Dec 9 06:45:01 2022 [283822.184089] Lustre: DEBUG MARKER: Fri Dec 9 06:50:01 2022 [284122.216438] Lustre: DEBUG MARKER: Fri Dec 9 06:55:01 2022 [284422.252005] Lustre: DEBUG MARKER: Fri Dec 9 07:00:01 2022 [284722.307067] Lustre: DEBUG MARKER: Fri Dec 9 07:05:01 2022 [285022.339324] Lustre: DEBUG MARKER: Fri Dec 9 07:10:01 2022 [285322.372643] Lustre: DEBUG MARKER: Fri Dec 9 07:15:01 2022 [285622.404518] Lustre: DEBUG MARKER: Fri Dec 9 07:20:01 2022 [285922.437289] Lustre: DEBUG MARKER: Fri Dec 9 07:25:01 2022 [286222.467985] Lustre: DEBUG MARKER: Fri Dec 9 07:30:01 2022 [286522.502087] Lustre: DEBUG MARKER: Fri Dec 9 07:35:01 2022 [286822.531685] Lustre: DEBUG MARKER: Fri Dec 9 07:40:01 2022 [287122.564222] Lustre: DEBUG MARKER: Fri Dec 9 07:45:01 2022 [287422.596951] Lustre: DEBUG MARKER: Fri Dec 9 07:50:01 2022 [287722.630099] Lustre: DEBUG MARKER: Fri Dec 9 07:55:01 2022 [288022.662189] Lustre: DEBUG MARKER: Fri Dec 9 08:00:01 2022 [288322.718964] Lustre: DEBUG MARKER: Fri Dec 9 08:05:01 2022 [288622.750330] Lustre: DEBUG MARKER: Fri Dec 9 08:10:01 2022 [288922.782780] Lustre: DEBUG MARKER: Fri Dec 9 08:15:01 2022 [289222.813903] Lustre: DEBUG MARKER: Fri Dec 9 08:20:01 2022 [289522.845898] Lustre: DEBUG MARKER: Fri Dec 9 08:25:01 2022 [289822.877069] Lustre: DEBUG MARKER: Fri Dec 9 08:30:01 2022 [290122.910036] Lustre: DEBUG MARKER: Fri Dec 9 08:35:01 2022 [290422.940824] Lustre: DEBUG MARKER: Fri Dec 9 08:40:01 2022 [290722.974351] Lustre: DEBUG MARKER: Fri Dec 9 08:45:01 2022 [291023.007208] Lustre: DEBUG MARKER: Fri Dec 9 08:50:01 2022 [291323.040207] Lustre: DEBUG MARKER: Fri Dec 9 08:55:01 2022 [291623.074350] Lustre: DEBUG MARKER: Fri Dec 9 09:00:01 2022 [291923.130123] Lustre: DEBUG MARKER: Fri Dec 9 09:05:01 2022 [292223.163271] Lustre: DEBUG MARKER: Fri Dec 9 09:10:01 2022 [292523.196493] Lustre: DEBUG MARKER: Fri Dec 9 09:15:01 2022 [292823.227804] Lustre: DEBUG MARKER: Fri Dec 9 09:20:01 2022 [293123.263442] Lustre: DEBUG MARKER: Fri Dec 9 09:25:01 2022 [293423.296539] Lustre: DEBUG MARKER: Fri Dec 9 09:30:01 2022 [293723.328003] Lustre: DEBUG MARKER: Fri Dec 9 09:35:01 2022 [294023.359767] Lustre: DEBUG MARKER: Fri Dec 9 09:40:01 2022 [294323.392873] Lustre: DEBUG MARKER: Fri Dec 9 09:45:01 2022 [294623.423490] Lustre: DEBUG MARKER: Fri Dec 9 09:50:01 2022 [294923.455463] Lustre: DEBUG MARKER: Fri Dec 9 09:55:01 2022 [295223.487226] Lustre: DEBUG MARKER: Fri Dec 9 10:00:02 2022 [295522.542566] Lustre: DEBUG MARKER: Fri Dec 9 10:05:01 2022 [295822.576140] Lustre: DEBUG MARKER: Fri Dec 9 10:10:01 2022 [296122.608517] Lustre: DEBUG MARKER: Fri Dec 9 10:15:01 2022 [296422.642356] Lustre: DEBUG MARKER: Fri Dec 9 10:20:01 2022 [296722.673005] Lustre: DEBUG MARKER: Fri Dec 9 10:25:01 2022 [297022.701714] Lustre: DEBUG MARKER: Fri Dec 9 10:30:01 2022 [297322.734168] Lustre: DEBUG MARKER: Fri Dec 9 10:35:01 2022 [297622.764888] Lustre: DEBUG MARKER: Fri Dec 9 10:40:01 2022 [297922.796309] Lustre: DEBUG MARKER: Fri Dec 9 10:45:01 2022 [298222.828025] Lustre: DEBUG MARKER: Fri Dec 9 10:50:01 2022 [298522.860960] Lustre: DEBUG MARKER: Fri Dec 9 10:55:01 2022 [298822.897703] Lustre: DEBUG MARKER: Fri Dec 9 11:00:01 2022 [299122.954159] Lustre: DEBUG MARKER: Fri Dec 9 11:05:01 2022 [299422.985727] Lustre: DEBUG MARKER: Fri Dec 9 11:10:01 2022 [299723.018657] Lustre: DEBUG MARKER: Fri Dec 9 11:15:01 2022 [300023.053487] Lustre: DEBUG MARKER: Fri Dec 9 11:20:01 2022 [300323.087333] Lustre: DEBUG MARKER: Fri Dec 9 11:25:01 2022 [300623.119047] Lustre: DEBUG MARKER: Fri Dec 9 11:30:01 2022 [300923.152736] Lustre: DEBUG MARKER: Fri Dec 9 11:35:01 2022 [301223.184455] Lustre: DEBUG MARKER: Fri Dec 9 11:40:01 2022 [301523.217761] Lustre: DEBUG MARKER: Fri Dec 9 11:45:01 2022 [301823.250562] Lustre: DEBUG MARKER: Fri Dec 9 11:50:01 2022 [302123.283837] Lustre: DEBUG MARKER: Fri Dec 9 11:55:01 2022 [302423.316467] Lustre: DEBUG MARKER: Fri Dec 9 12:00:01 2022 [302723.368202] Lustre: DEBUG MARKER: Fri Dec 9 12:05:01 2022 [303023.396058] Lustre: DEBUG MARKER: Fri Dec 9 12:10:01 2022 [303323.426974] Lustre: DEBUG MARKER: Fri Dec 9 12:15:01 2022 [303623.457201] Lustre: DEBUG MARKER: Fri Dec 9 12:20:01 2022 [303923.489525] Lustre: DEBUG MARKER: Fri Dec 9 12:25:01 2022 [304223.519999] Lustre: DEBUG MARKER: Fri Dec 9 12:30:01 2022 [304523.551281] Lustre: DEBUG MARKER: Fri Dec 9 12:35:01 2022 [304823.582466] Lustre: DEBUG MARKER: Fri Dec 9 12:40:01 2022 [305123.612713] Lustre: DEBUG MARKER: Fri Dec 9 12:45:01 2022 [305423.640712] Lustre: DEBUG MARKER: Fri Dec 9 12:50:01 2022 [305723.670724] Lustre: DEBUG MARKER: Fri Dec 9 12:55:01 2022 [306023.703142] Lustre: DEBUG MARKER: Fri Dec 9 13:00:01 2022 [306323.753582] Lustre: DEBUG MARKER: Fri Dec 9 13:05:01 2022 [306623.783879] Lustre: DEBUG MARKER: Fri Dec 9 13:10:01 2022 [306923.814104] Lustre: DEBUG MARKER: Fri Dec 9 13:15:01 2022 [307223.842622] Lustre: DEBUG MARKER: Fri Dec 9 13:20:01 2022 [307523.872756] Lustre: DEBUG MARKER: Fri Dec 9 13:25:01 2022 [307823.902036] Lustre: DEBUG MARKER: Fri Dec 9 13:30:01 2022 [308123.933910] Lustre: DEBUG MARKER: Fri Dec 9 13:35:01 2022 [308423.963779] Lustre: DEBUG MARKER: Fri Dec 9 13:40:01 2022 [308723.993825] Lustre: DEBUG MARKER: Fri Dec 9 13:45:01 2022 [309023.024696] Lustre: DEBUG MARKER: Fri Dec 9 13:50:01 2022 [309184.968342] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [309184.970090] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 2, rc: 42 [309235.970192] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [309235.973308] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [309285.972132] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [309285.975377] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [309291.972335] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [309291.975707] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [309303.972873] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [309303.976038] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 1 previous similar message [309303.977756] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [309303.980211] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 1 previous similar message [309321.973514] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [309321.976797] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 2 previous similar messages [309321.978581] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 41 [309321.981130] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 2 previous similar messages [309323.055385] Lustre: DEBUG MARKER: Fri Dec 9 13:55:01 2022 [309356.974871] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [309356.976752] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 3 previous similar messages [309356.978595] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [309356.981234] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 3 previous similar messages [309422.977367] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [309422.980733] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 10 previous similar messages [309422.982597] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [309422.985266] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 10 previous similar messages [309553.982445] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [309553.986001] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 20 previous similar messages [309553.987864] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [309553.990543] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 20 previous similar messages [309623.086541] Lustre: DEBUG MARKER: Fri Dec 9 14:00:01 2022 [309812.992245] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds [309812.995390] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 40 previous similar messages [309812.997295] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (7): c: 0, oc: 0, rc: 42 [309813.000057] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 40 previous similar messages [309923.140343] Lustre: DEBUG MARKER: Fri Dec 9 14:05:01 2022 [310223.171761] Lustre: DEBUG MARKER: Fri Dec 9 14:10:01 2022 [310330.012033] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [310330.015522] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 81 previous similar messages [310330.017421] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [310330.020142] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 81 previous similar messages [310463.657229] LNet: 5567:0:(o2iblnd_cb.c:2565:kiblnd_passive_connect()) Can't accept conn from AA.AA.ID21@o2ibxx (version 12): max_frags 257 too large (256 wanted) [310523.202417] Lustre: DEBUG MARKER: Fri Dec 9 14:15:01 2022 [310597.022121] LNet: 17020:0:(o2iblnd_cb.c:3429:kiblnd_check_conns()) Timed out tx for AA.AA.ID22@o2ibxx: 1 seconds [310823.232495] Lustre: DEBUG MARKER: Fri Dec 9 14:20:01 2022 [310936.035114] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [310936.038246] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 95 previous similar messages [310936.040180] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [310936.042963] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 95 previous similar messages [311123.267962] Lustre: DEBUG MARKER: Fri Dec 9 14:25:01 2022 [311207.702358] LNet: 14494:0:(o2iblnd_cb.c:2565:kiblnd_passive_connect()) Can't accept conn from AA.AA.ID21@o2ibxx (version 12): max_frags 257 too large (256 wanted) [311303.049027] LNet: 17020:0:(o2iblnd_cb.c:3429:kiblnd_check_conns()) Timed out tx for AA.AA.ID23@o2ibxx: 4 seconds [311303.052486] LNet: 17020:0:(o2iblnd_cb.c:3429:kiblnd_check_conns()) Skipped 14 previous similar messages [311423.299912] Lustre: DEBUG MARKER: Fri Dec 9 14:30:01 2022 [311498.728077] LNet: 14494:0:(o2iblnd_cb.c:2565:kiblnd_passive_connect()) Can't accept conn from AA.AA.ID21@o2ibxx (version 12): max_frags 257 too large (256 wanted) [311542.058059] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [311542.061510] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 95 previous similar messages [311542.063586] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [311542.066589] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 95 previous similar messages [311723.334992] Lustre: DEBUG MARKER: Fri Dec 9 14:35:01 2022 [311851.752191] LNet: 14494:0:(o2iblnd_cb.c:2565:kiblnd_passive_connect()) Can't accept conn from AA.AA.ID21@o2ibxx (version 12): max_frags 257 too large (256 wanted) [312023.367667] Lustre: DEBUG MARKER: Fri Dec 9 14:40:01 2022 [312148.081309] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [312148.085086] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 95 previous similar messages [312148.087065] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [312148.089923] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 95 previous similar messages [312323.402099] Lustre: DEBUG MARKER: Fri Dec 9 14:45:01 2022 [312623.433291] Lustre: DEBUG MARKER: Fri Dec 9 14:50:01 2022 [312754.104303] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [312754.106289] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 95 previous similar messages [312754.108224] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [312754.111064] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 95 previous similar messages [312923.468649] Lustre: DEBUG MARKER: Fri Dec 9 14:55:01 2022 [313190.839822] LNet: 15808:0:(o2iblnd_cb.c:2565:kiblnd_passive_connect()) Can't accept conn from AA.AA.ID21@o2ibxx (version 12): max_frags 257 too large (256 wanted) [313223.501533] Lustre: DEBUG MARKER: Fri Dec 9 15:00:01 2022 [313360.127131] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [313360.130626] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 95 previous similar messages [313360.132593] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [313360.135451] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 95 previous similar messages [313523.564634] Lustre: DEBUG MARKER: Fri Dec 9 15:05:01 2022 [313823.595627] Lustre: DEBUG MARKER: Fri Dec 9 15:10:01 2022 [313871.880361] LNet: 27716:0:(o2iblnd_cb.c:2565:kiblnd_passive_connect()) Can't accept conn from AA.AA.ID21@o2ibxx (version 12): max_frags 257 too large (256 wanted) [313966.149989] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [313966.153290] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 95 previous similar messages [313966.155235] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [313966.158062] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 95 previous similar messages [314123.659047] Lustre: DEBUG MARKER: Fri Dec 9 15:15:01 2022 [314215.145286] LNet: 11618:0:(o2iblnd_cb.c:2565:kiblnd_passive_connect()) Can't accept conn from AA.AA.ID24@o2ibxx (version 12): max_frags 257 too large (256 wanted) [314423.747500] Lustre: DEBUG MARKER: Fri Dec 9 15:20:01 2022 [314572.172882] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [314572.176420] LNetError: 17020:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Skipped 95 previous similar messages [314572.178784] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with BB.BB.ID17@o2ibyy (6): c: 0, oc: 0, rc: 42 [314572.182082] LNetError: 17020:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Skipped 95 previous similar messages [314590.492274] kworker/u20:0 invoked oom-killer: gfp_mask=0x80d0, order=0, oom_score_adj=0 [314590.494459] kworker/u20:0 cpuset=/ mems_allowed=0-1 [314590.495806] CPU: 3 PID: 7919 Comm: kworker/u20:0 Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314590.498638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314590.501485] Workqueue: rdma_cm cma_work_handler [rdma_cm] [314590.502883] Call Trace: [314590.503928] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314590.505272] [<ffffffff95b81668>] dump_header+0x90/0x229 [314590.506629] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314590.508023] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314590.509396] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314590.510795] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314590.512213] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314590.513557] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314590.514992] [<ffffffff9543346f>] dma_generic_alloc_coherent+0x8f/0x140 [314590.516439] [<ffffffff9546eb01>] x86_swiotlb_alloc_coherent+0x21/0x50 [314590.517980] [<ffffffffc036f3a9>] mlx5_dma_zalloc_coherent_node+0xb9/0x110 [mlx5_core] [314590.519602] [<ffffffffc036f9ba>] mlx5_frag_buf_alloc_node+0xfa/0x310 [mlx5_core] [314590.521147] [<ffffffffc036f589>] ? mlx5_db_alloc_node+0x69/0x1a0 [mlx5_core] [314590.522663] [<ffffffffc05516d3>] mlx5_ib_create_cq+0x3e3/0x680 [mlx5_ib] [314590.524120] [<ffffffff95628ceb>] ? __kmalloc+0x1eb/0x230 [314590.525433] [<ffffffffc06dc191>] __ib_create_cq+0xa1/0xf0 [ib_core] [314590.526853] [<ffffffffc0936e90>] ? kiblnd_cq_completion+0x160/0x160 [ko2iblnd] [314590.528350] [<ffffffffc092563a>] kiblnd_create_conn+0xcba/0x1a00 [ko2iblnd] [314590.529805] [<ffffffffc093514e>] kiblnd_cm_callback+0x145e/0x23a0 [ko2iblnd] [314590.531260] [<ffffffffc07b9d81>] cma_work_handler+0xa1/0xe0 [rdma_cm] [314590.532657] [<ffffffff954bdfbf>] process_one_work+0x17f/0x440 [314590.533966] [<ffffffff954bf0d6>] worker_thread+0x126/0x3c0 [314590.535234] [<ffffffff954befb0>] ? manage_workers.isra.26+0x2a0/0x2a0 [314590.536600] [<ffffffff954c5f91>] kthread+0xd1/0xe0 [314590.537759] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314590.539062] [<ffffffff95b99ddd>] ret_from_fork_nospec_begin+0x7/0x21 [314590.540399] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314590.541692] Mem-Info: [314590.542513] active_anon:169852 inactive_anon:22463 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40556 slab_unreclaimable:110515 mapped:17361 shmem:170193 pagetables:1250 bounce:0 free:91298 free_pcp:2145 free_cma:0 [314590.549447] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314590.556412] lowmem_reserve[]: 0 2607 24634 24634 [314590.557600] Node 0 DMA32 free:101500kB min:13744kB low:17180kB high:20616kB active_anon:88kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6468kB slab_unreclaimable:26260kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:2036kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:15 all_unreclaimable? yes [314590.564918] lowmem_reserve[]: 0 0 22026 22026 [314590.566268] Node 0 Normal free:115872kB min:116096kB low:145120kB high:174144kB active_anon:56508kB inactive_anon:41092kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:30964kB shmem:41276kB slab_reclaimable:81568kB slab_unreclaimable:198668kB kernel_stack:2288kB pagetables:2388kB unstable:0kB bounce:0kB free_pcp:3304kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314590.576367] lowmem_reserve[]: 0 0 0 0 [314590.577685] Node 1 Normal free:132008kB min:132216kB low:165268kB high:198324kB active_anon:622812kB inactive_anon:48760kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:38480kB shmem:639496kB slab_reclaimable:74188kB slab_unreclaimable:217100kB kernel_stack:2784kB pagetables:2612kB unstable:0kB bounce:0kB free_pcp:3240kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:38 all_unreclaimable? yes [314590.588171] lowmem_reserve[]: 0 0 0 0 [314590.589551] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314590.593392] Node 0 DMA32: 70*4kB (UEM) 57*8kB (UEM) 28*16kB (UEM) 17*32kB (UM) 15*64kB (UM) 7*128kB (UEM) 8*256kB (EM) 7*512kB (UM) 6*1024kB (EM) 4*2048kB (UM) 19*4096kB (M) = 101376kB [314590.597876] Node 0 Normal: 2381*4kB (UEM) 1842*8kB (UEM) 1131*16kB (EM) 678*32kB (UEM) 397*64kB (EM) 163*128kB (UEM) 5*256kB (M) 1*512kB (U) 0*1024kB 2*2048kB (U) 0*4096kB = 116212kB [314590.602308] Node 1 Normal: 3055*4kB (EM) 1992*8kB (EM) 1396*16kB (UM) 1051*32kB (UEM) 738*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 131356kB [314590.606481] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314590.608659] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314590.610783] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314590.612927] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314590.615028] 170174 total pagecache pages [314590.616418] 0 pages in swap cache [314590.617719] Swap cache stats: add 0, delete 0, find 0/0 [314590.619310] Free swap = 0kB [314590.620558] Total swap = 0kB [314590.621806] 13055868 pages RAM [314590.623052] 0 pages HighMem/MovableOnly [314590.624421] 323772 pages reserved [314590.625710] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314590.627743] [ 602] 0 602 48862 27680 99 0 0 systemd-journal [314590.629906] [ 618] 0 618 49643 96 30 0 0 lvmetad [314590.631955] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314590.634080] [ 801] 0 801 13883 100 29 0 -1000 auditd [314590.636101] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314590.638150] [ 861] 0 861 13212 203 30 0 0 smartd [314590.640170] [ 872] 0 872 11042 91 29 0 0 qemu-ga [314590.642172] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314590.644208] [ 898] 38 898 7457 127 19 0 0 ntpd [314590.646146] [ 916] 0 916 6596 77 17 0 0 systemd-logind [314590.648211] [ 1486] 0 1486 25746 531 49 0 0 dhclient [314590.650186] [ 1819] 997 1819 216185 1865 51 0 0 lustre_exporter [314590.652231] [ 1820] 997 1820 238088 1538 58 0 0 pushprox-client [314590.654258] [ 1821] 997 1821 253782 3960 62 0 0 node_exporter [314590.656238] [ 1823] 99 1823 14304 369 30 0 0 dnsmasq [314590.658131] [ 1824] 89 1824 56604 2425 62 0 0 python [314590.659997] [ 1879] 89 1879 54555 2420 58 0 0 python [314590.661844] [ 1949] 0 1949 22964 264 45 0 0 master [314590.663681] [ 1951] 89 1951 23007 255 48 0 0 qmgr [314590.665466] [ 2202] 0 2202 28244 258 57 0 -1000 sshd [314590.667255] [ 2207] 0 2207 63618 394 31 0 0 rsyslogd [314590.669089] [ 2208] 0 2208 27552 31 11 0 0 agetty [314590.670873] [ 2209] 0 2209 6477 50 18 0 0 atd [314590.672605] [ 2210] 0 2210 27552 31 8 0 0 agetty [314590.674360] [ 2214] 0 2214 31597 155 18 0 0 crond [314590.676084] [16982] 0 16982 29083 302 14 0 0 net_device_wait [314590.677952] [16983] 0 16983 29083 302 12 0 0 net_device_wait [314590.679795] [17164] 0 17164 372278 5439 120 0 0 ebpf_exporter [314590.681684] [21402] 0 21402 38696 325 77 0 0 sshd [314590.683371] [21404] 0 21404 28953 165 11 0 0 bash [314590.685025] [27616] 89 27616 22990 257 46 0 0 pickup [314590.686702] [11752] 0 11752 27014 22 9 0 0 sleep [314590.688337] [11753] 0 11753 27014 23 8 0 0 sleep [314590.689967] Out of memory: Kill process 602 (systemd-journal) score 2 or sacrifice child [314590.691621] Killed process 602 (systemd-journal), UID 0, total-vm:195448kB, anon-rss:332kB, file-rss:0kB, shmem-rss:110388kB [314590.697669] qemu-ga invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [314590.700155] qemu-ga cpuset=/ mems_allowed=0-1 [314590.701058] CPU: 4 PID: 872 Comm: qemu-ga Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314590.703169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314590.705395] Call Trace: [314590.706119] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314590.707164] [<ffffffff95b81668>] dump_header+0x90/0x229 [314590.708235] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314590.709337] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314590.710437] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314590.711566] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314590.712721] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314590.713806] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314590.714987] [<ffffffff956193b8>] alloc_pages_current+0x98/0x110 [314590.716128] [<ffffffff955be057>] __page_cache_alloc+0x97/0xb0 [314590.717246] [<ffffffff955c1000>] filemap_fault+0x270/0x420 [314590.718425] [<ffffffffc05fc91e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [314590.719644] [<ffffffffc05fcb1c>] xfs_filemap_fault+0x2c/0x30 [xfs] [314590.721093] [<ffffffff955ee7da>] __do_fault.isra.61+0x8a/0x100 [314590.722494] [<ffffffff955eed8c>] do_read_fault.isra.63+0x4c/0x1b0 [314590.723916] [<ffffffff9542b621>] ? __switch_to+0x151/0x580 [314590.725244] [<ffffffff955f65d0>] handle_mm_fault+0xa20/0xfb0 [314590.726601] [<ffffffff95b8c070>] ? __schedule+0x320/0x680 [314590.727907] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314590.729257] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314590.730650] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314590.732031] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314590.733378] Mem-Info: [314590.734197] active_anon:179853 inactive_anon:12462 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40556 slab_unreclaimable:110515 mapped:0 shmem:170193 pagetables:1250 bounce:0 free:91258 free_pcp:2192 free_cma:0 [314590.741995] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314590.750471] lowmem_reserve[]: 0 2607 24634 24634 [314590.751827] Node 0 DMA32 free:101500kB min:13744kB low:17180kB high:20616kB active_anon:88kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6468kB slab_unreclaimable:26260kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:2048kB local_pcp:656kB free_cma:0kB writeback_tmp:0kB pages_scanned:24 all_unreclaimable? yes [314590.760962] lowmem_reserve[]: 0 0 22026 22026 [314590.762457] Node 0 Normal free:115792kB min:116096kB low:145120kB high:174144kB active_anon:71108kB inactive_anon:26492kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:41276kB slab_reclaimable:81568kB slab_unreclaimable:198668kB kernel_stack:2288kB pagetables:2388kB unstable:0kB bounce:0kB free_pcp:3216kB local_pcp:248kB free_cma:0kB writeback_tmp:0kB pages_scanned:6 all_unreclaimable? yes [314590.772678] lowmem_reserve[]: 0 0 0 0 [314590.773937] Node 1 Normal free:131928kB min:132216kB low:165268kB high:198324kB active_anon:648216kB inactive_anon:23356kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:639496kB slab_reclaimable:74188kB slab_unreclaimable:217100kB kernel_stack:2784kB pagetables:2612kB unstable:0kB bounce:0kB free_pcp:3504kB local_pcp:348kB free_cma:0kB writeback_tmp:0kB pages_scanned:27 all_unreclaimable? yes [314590.784216] lowmem_reserve[]: 0 0 0 0 [314590.785563] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314590.789294] Node 0 DMA32: 70*4kB (UEM) 57*8kB (UEM) 28*16kB (UEM) 17*32kB (UM) 15*64kB (UM) 7*128kB (UEM) 8*256kB (EM) 7*512kB (UM) 6*1024kB (EM) 4*2048kB (UM) 19*4096kB (M) = 101376kB [314590.793710] Node 0 Normal: 2371*4kB (EM) 1842*8kB (UEM) 1131*16kB (EM) 677*32kB (UEM) 397*64kB (EM) 162*128kB (EM) 5*256kB (M) 1*512kB (U) 0*1024kB 2*2048kB (U) 0*4096kB = 116012kB [314590.798112] Node 1 Normal: 3070*4kB (EM) 1996*8kB (EM) 1395*16kB (UM) 1049*32kB (UEM) 736*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 131240kB [314590.802199] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314590.804348] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314590.806463] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314590.808612] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314590.810707] 170174 total pagecache pages [314590.812095] 0 pages in swap cache [314590.813394] Swap cache stats: add 0, delete 0, find 0/0 [314590.814990] Free swap = 0kB [314590.816215] Total swap = 0kB [314590.817436] 13055868 pages RAM [314590.818692] 0 pages HighMem/MovableOnly [314590.820052] 323772 pages reserved [314590.821334] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314590.823348] [ 618] 0 618 49643 96 30 0 0 lvmetad [314590.825398] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314590.827536] [ 801] 0 801 13883 100 29 0 -1000 auditd [314590.829567] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314590.831633] [ 861] 0 861 13212 203 30 0 0 smartd [314590.833640] [ 872] 0 872 11042 91 29 0 0 qemu-ga [314590.835664] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314590.837718] [ 898] 38 898 7457 127 19 0 0 ntpd [314590.839662] [ 916] 0 916 6596 77 17 0 0 systemd-logind [314590.841729] [ 1486] 0 1486 25746 531 49 0 0 dhclient [314590.843708] [ 1819] 997 1819 216185 1865 51 0 0 lustre_exporter [314590.845797] [ 1820] 997 1820 238088 1538 58 0 0 pushprox-client [314590.847868] [ 1821] 997 1821 253782 3960 62 0 0 node_exporter [314590.849924] [ 1823] 99 1823 14304 369 30 0 0 dnsmasq [314590.877011] [ 1824] 89 1824 56604 2425 62 0 0 python [314590.899472] [ 1879] 89 1879 54555 2420 58 0 0 python [314590.916468] [ 1949] 0 1949 22964 264 45 0 0 master [314590.926931] [ 1951] 89 1951 23007 255 48 0 0 qmgr [314590.928786] [ 2202] 0 2202 28244 258 57 0 -1000 sshd [314590.930253] [ 2207] 0 2207 63618 394 31 0 0 rsyslogd [314590.931768] [ 2208] 0 2208 27552 31 11 0 0 agetty [314590.933228] [ 2209] 0 2209 6477 50 18 0 0 atd [314590.934651] [ 2210] 0 2210 27552 31 8 0 0 agetty [314590.936087] [ 2214] 0 2214 31597 155 18 0 0 crond [314590.937505] [16982] 0 16982 29083 302 14 0 0 net_device_wait [314590.939385] [16983] 0 16983 29083 302 12 0 0 net_device_wait [314590.941262] [17164] 0 17164 372278 5439 120 0 0 ebpf_exporter [314590.943124] [21402] 0 21402 38696 325 77 0 0 sshd [314590.944811] [21404] 0 21404 28953 165 11 0 0 bash [314590.946479] [27616] 89 27616 22990 257 46 0 0 pickup [314590.948168] [11754] 0 11754 1168 13 7 0 0 systemd-cgroups [314590.949957] Out of memory: Kill process 17164 (ebpf_exporter) score 0 or sacrifice child [314590.951680] Killed process 17164 (ebpf_exporter), UID 0, total-vm:1489112kB, anon-rss:21756kB, file-rss:0kB, shmem-rss:0kB [314590.954960] kworker/u20:0 invoked oom-killer: gfp_mask=0x80d0, order=0, oom_score_adj=0 [314590.956828] kworker/u20:0 cpuset=/ mems_allowed=0-1 [314590.963350] CPU: 2 PID: 7919 Comm: kworker/u20:0 Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314590.983231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314591.003694] Workqueue: rdma_cm cma_work_handler [rdma_cm] [314591.012843] Call Trace: [314591.017958] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314591.025446] [<ffffffff95b81668>] dump_header+0x90/0x229 [314591.031611] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314591.039202] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314591.048086] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314591.057334] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314591.067868] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314591.077399] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314591.087360] [<ffffffff9543346f>] dma_generic_alloc_coherent+0x8f/0x140 [314591.098020] [<ffffffff9546eb01>] x86_swiotlb_alloc_coherent+0x21/0x50 [314591.109223] [<ffffffffc036f3a9>] mlx5_dma_zalloc_coherent_node+0xb9/0x110 [mlx5_core] [314591.127113] [<ffffffffc036f9ba>] mlx5_frag_buf_alloc_node+0xfa/0x310 [mlx5_core] [314591.142262] [<ffffffffc0585e8e>] _create_kernel_qp.isra.66+0x41c/0x80e [mlx5_ib] [314591.154562] [<ffffffff956eb1a3>] ? __create_file+0xf3/0x2c0 [314591.161139] [<ffffffffc05863e0>] create_kernel_qp.isra.76+0x160/0x52a [mlx5_ib] [314591.172119] [<ffffffffc056d2a5>] mlx5_ib_create_qp+0x13c5/0x1570 [mlx5_ib] [314591.173558] LNet: 17020:0:(o2iblnd_cb.c:3429:kiblnd_check_conns()) Timed out tx for AA.AA.ID25@o2ibxx: 832 seconds [314591.194347] [<ffffffffc032e3c2>] ? backport___xa_alloc+0xc2/0x190 [mlx_compat] [314591.201541] [<ffffffffc06dbcd6>] ib_create_qp+0xa6/0x3f0 [ib_core] [314591.206321] [<ffffffffc07b62f1>] rdma_create_qp+0x41/0xc0 [rdma_cm] [314591.209172] [<ffffffffc092581c>] kiblnd_create_conn+0xe9c/0x1a00 [ko2iblnd] [314591.212076] [<ffffffffc093514e>] kiblnd_cm_callback+0x145e/0x23a0 [ko2iblnd] [314591.215747] [<ffffffffc07b9d81>] cma_work_handler+0xa1/0xe0 [rdma_cm] [314591.219034] [<ffffffff954bdfbf>] process_one_work+0x17f/0x440 [314591.222198] [<ffffffff954bf0d6>] worker_thread+0x126/0x3c0 [314591.227741] [<ffffffff954befb0>] ? manage_workers.isra.26+0x2a0/0x2a0 [314591.229890] [<ffffffff954c5f91>] kthread+0xd1/0xe0 [314591.231446] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314591.233142] [<ffffffff95b99ddd>] ret_from_fork_nospec_begin+0x7/0x21 [314591.246015] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314591.251256] Mem-Info: [314591.252290] active_anon:173950 inactive_anon:12376 isolated_anon:0 active_file:105 inactive_file:539 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40560 slab_unreclaimable:110531 mapped:325 shmem:170193 pagetables:1046 bounce:0 free:103977 free_pcp:1964 free_cma:0 [314591.262855] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314591.273818] lowmem_reserve[]: 0 2607 24634 24634 [314591.275502] Node 0 DMA32 free:105196kB min:13744kB low:17180kB high:20616kB active_anon:88kB inactive_anon:0kB active_file:688kB inactive_file:3092kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:1536kB shmem:0kB slab_reclaimable:6468kB slab_unreclaimable:26260kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:2052kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [314591.290393] lowmem_reserve[]: 0 0 22026 22026 [314591.292479] Node 0 Normal free:143828kB min:116096kB low:145120kB high:174144kB active_anon:51488kB inactive_anon:26256kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:41276kB slab_reclaimable:81568kB slab_unreclaimable:198668kB kernel_stack:2288kB pagetables:1812kB unstable:0kB bounce:0kB free_pcp:2688kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:3463 all_unreclaimable? yes [314591.306952] lowmem_reserve[]: 0 0 0 0 [314591.308629] Node 1 Normal free:151132kB min:132216kB low:165268kB high:198324kB active_anon:644224kB inactive_anon:23248kB active_file:8kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:116kB shmem:639496kB slab_reclaimable:74204kB slab_unreclaimable:217164kB kernel_stack:2800kB pagetables:2372kB unstable:0kB bounce:0kB free_pcp:2988kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:11 all_unreclaimable? no [314591.319946] lowmem_reserve[]: 0 0 0 0 [314591.321525] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314591.325692] Node 0 DMA32: 2*4kB (UM) 17*8kB (UE) 3*16kB (UEM) 5*32kB (UE) 7*64kB (U) 5*128kB (UE) 5*256kB (UEM) 10*512kB (UM) 7*1024kB (UEM) 4*2048kB (UM) 20*4096kB (M) = 105120kB [314591.330786] Node 0 Normal: 2594*4kB (UEM) 1919*8kB (UEM) 1211*16kB (UEM) 743*32kB (UEM) 437*64kB (UEM) 196*128kB (UEM) 22*256kB (UM) 12*512kB (UM) 6*1024kB (UM) 2*2048kB (U) 0*4096kB = 143952kB [314591.335853] Node 1 Normal: 3418*4kB (UEM) 2084*8kB (UEM) 1456*16kB (UM) 1156*32kB (UEM) 785*64kB (UM) 19*128kB (UM) 19*256kB (U) 6*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 151240kB [314591.341572] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314591.344619] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314591.347599] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314591.350361] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314591.353477] 170834 total pagecache pages [314591.355249] 0 pages in swap cache [314591.356816] Swap cache stats: add 0, delete 0, find 0/0 [314591.358958] Free swap = 0kB [314591.366748] Total swap = 0kB [314591.371072] 13055868 pages RAM [314591.373697] 0 pages HighMem/MovableOnly [314591.377681] 323772 pages reserved [314591.379042] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314591.384268] [ 618] 0 618 49643 96 30 0 0 lvmetad [314591.391505] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314591.405521] [ 801] 0 801 13883 110 29 0 -1000 auditd [314591.410613] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314591.415094] [ 861] 0 861 13212 203 30 0 0 smartd [314591.417219] [ 872] 0 872 11042 91 29 0 0 qemu-ga [314591.419375] [ 878] 81 878 14536 232 32 0 -900 dbus-daemon [314591.421671] [ 898] 38 898 7457 127 19 0 0 ntpd [314591.423727] [ 916] 0 916 6596 119 17 0 0 systemd-logind [314591.425926] [ 1486] 0 1486 25746 531 49 0 0 dhclient [314591.428218] [ 1819] 997 1819 216185 1865 51 0 0 lustre_exporter [314591.430775] [ 1820] 997 1820 238088 1538 58 0 0 pushprox-client [314591.433450] [ 1821] 997 1821 253782 3960 62 0 0 node_exporter [314591.436104] [ 1823] 99 1823 14304 369 30 0 0 dnsmasq [314591.438672] [ 1824] 89 1824 56604 2425 62 0 0 python [314591.441229] [ 1879] 89 1879 54555 2420 58 0 0 python [314591.443729] [ 1949] 0 1949 22964 264 45 0 0 master [314591.446369] [ 1951] 89 1951 23007 255 48 0 0 qmgr [314591.449756] [ 2202] 0 2202 28244 258 57 0 -1000 sshd [314591.453423] [ 2207] 0 2207 63618 394 31 0 0 rsyslogd [314591.457140] [ 2208] 0 2208 27552 31 11 0 0 agetty [314591.459621] [ 2209] 0 2209 6477 50 18 0 0 atd [314591.461702] [ 2210] 0 2210 27552 31 8 0 0 agetty [314591.463887] [ 2214] 0 2214 31597 155 18 0 0 crond [314591.466433] [16982] 0 16982 29083 387 14 0 0 net_device_wait [314591.468610] [16983] 0 16983 29083 371 12 0 0 net_device_wait [314591.471049] [21402] 0 21402 38696 325 77 0 0 sshd [314591.473011] [21404] 0 21404 28953 165 11 0 0 bash [314591.476454] [27616] 89 27616 22990 257 46 0 0 pickup [314591.479635] [11759] 0 11759 27014 131 9 0 0 sleep [314591.485657] [11760] 0 11760 27014 130 9 0 0 sleep [314591.489944] [11762] 0 11762 2437 29 10 0 0 systemd-journal [314591.496246] Out of memory: Kill process 1821 (node_exporter) score 0 or sacrifice child [314591.502958] Killed process 1821 (node_exporter), UID 997, total-vm:1015128kB, anon-rss:15840kB, file-rss:0kB, shmem-rss:0kB [314592.000266] systemd-journald[11762]: File /run/log/journal/51f91475367e4abeacabe8518b37dc38/system.journal corrupted or uncleanly shut down, renaming and replacing. [314595.910004] net_device_wait invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [314595.913560] net_device_wait cpuset=/ mems_allowed=0-1 [314595.914663] CPU: 6 PID: 11778 Comm: net_device_wait Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314595.916936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314595.919269] Call Trace: [314595.920081] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314595.921219] [<ffffffff95b81668>] dump_header+0x90/0x229 [314595.922376] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314595.923584] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314595.924765] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314595.925903] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314595.927083] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314595.928193] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314595.929394] [<ffffffff956193b8>] alloc_pages_current+0x98/0x110 [314595.930559] [<ffffffff955be057>] __page_cache_alloc+0x97/0xb0 [314595.931704] [<ffffffff955c1000>] filemap_fault+0x270/0x420 [314595.932844] [<ffffffffc05fc91e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [314595.934090] [<ffffffffc05fcb1c>] xfs_filemap_fault+0x2c/0x30 [xfs] [314595.935292] [<ffffffff955ee7da>] __do_fault.isra.61+0x8a/0x100 [314595.936445] [<ffffffff955eed8c>] do_read_fault.isra.63+0x4c/0x1b0 [314595.937623] [<ffffffff955f65d0>] handle_mm_fault+0xa20/0xfb0 [314595.938767] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314595.939902] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314595.941058] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314595.942207] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314595.943336] Mem-Info: [314595.944077] active_anon:169929 inactive_anon:14408 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40560 slab_unreclaimable:110563 mapped:853 shmem:172240 pagetables:996 bounce:0 free:91337 free_pcp:488 free_cma:0 [314595.950735] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314595.957905] lowmem_reserve[]: 0 2607 24634 24634 [314595.959160] Node 0 DMA32 free:101608kB min:13744kB low:17180kB high:20616kB active_anon:88kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26252kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:1180kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:42 all_unreclaimable? yes [314595.966502] lowmem_reserve[]: 0 0 22026 22026 [314595.967716] Node 0 Normal free:115928kB min:116096kB low:145120kB high:174144kB active_anon:39320kB inactive_anon:26244kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:36kB shmem:41272kB slab_reclaimable:81576kB slab_unreclaimable:198712kB kernel_stack:1776kB pagetables:1632kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314595.975999] lowmem_reserve[]: 0 0 0 0 [314595.977132] Node 1 Normal free:132000kB min:132216kB low:165268kB high:198324kB active_anon:640308kB inactive_anon:31388kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:3872kB shmem:647688kB slab_reclaimable:74180kB slab_unreclaimable:217256kB kernel_stack:2656kB pagetables:2352kB unstable:0kB bounce:0kB free_pcp:772kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:38 all_unreclaimable? yes [314595.985482] lowmem_reserve[]: 0 0 0 0 [314595.986746] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314595.990050] Node 0 DMA32: 27*4kB (EM) 26*8kB (UEM) 4*16kB (UM) 4*32kB (UEM) 8*64kB (UM) 15*128kB (UEM) 7*256kB (EM) 4*512kB (M) 4*1024kB (EM) 4*2048kB (UM) 20*4096kB (M) = 100988kB [314595.993720] Node 0 Normal: 2292*4kB (EM) 1719*8kB (EM) 1119*16kB (UEM) 693*32kB (UEM) 407*64kB (UEM) 184*128kB (UEM) 13*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115928kB [314595.997334] Node 1 Normal: 1126*4kB (EM) 2077*8kB (EM) 1428*16kB (EM) 1081*32kB (EM) 751*64kB (UM) 12*128kB (M) 1*256kB (U) 1*512kB (U) 3*1024kB (U) 0*2048kB 0*4096kB = 132000kB [314596.001061] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314596.002857] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314596.004608] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314596.006388] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314596.008133] 172284 total pagecache pages [314596.009357] 0 pages in swap cache [314596.010510] Swap cache stats: add 0, delete 0, find 0/0 [314596.011891] Free swap = 0kB [314596.012993] Total swap = 0kB [314596.014101] 13055868 pages RAM [314596.015218] 0 pages HighMem/MovableOnly [314596.016424] 323772 pages reserved [314596.017564] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314596.019244] [ 618] 0 618 49643 96 30 0 0 lvmetad [314596.020960] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314596.022741] [ 801] 0 801 13883 110 29 0 -1000 auditd [314596.024433] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314596.026161] [ 861] 0 861 13212 203 30 0 0 smartd [314596.027843] [ 872] 0 872 11042 91 29 0 0 qemu-ga [314596.029516] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314596.031218] [ 898] 38 898 7457 127 19 0 0 ntpd [314596.032836] [ 916] 0 916 6596 77 17 0 0 systemd-logind [314596.034539] [ 1486] 0 1486 25746 531 49 0 0 dhclient [314596.036172] [ 1819] 997 1819 216185 1865 51 0 0 lustre_exporter [314596.037876] [ 1820] 997 1820 238088 1566 58 0 0 pushprox-client [314596.039573] [ 1823] 99 1823 14304 369 30 0 0 dnsmasq [314596.041237] [ 1824] 89 1824 56604 2425 62 0 0 python [314596.042848] [ 1879] 89 1879 54555 2420 58 0 0 python [314596.044383] [ 1949] 0 1949 22964 264 45 0 0 master [314596.045917] [ 1951] 89 1951 23007 255 48 0 0 qmgr [314596.047405] [ 2202] 0 2202 28244 258 57 0 -1000 sshd [314596.048883] [ 2207] 0 2207 63618 394 31 0 0 rsyslogd [314596.050383] [ 2208] 0 2208 27552 31 11 0 0 agetty [314596.051870] [ 2209] 0 2209 6477 50 18 0 0 atd [314596.053310] [ 2210] 0 2210 27552 31 8 0 0 agetty [314596.054766] [ 2214] 0 2214 31597 155 18 0 0 crond [314596.056237] [16982] 0 16982 29083 302 14 0 0 net_device_wait [314596.058010] [16983] 0 16983 29083 302 12 0 0 net_device_wait [314596.059663] [21402] 0 21402 38696 325 77 0 0 sshd [314596.061164] [21404] 0 21404 28953 165 11 0 0 bash [314596.062644] [27616] 89 27616 22990 257 46 0 0 pickup [314596.064136] [11762] 0 11762 9851 1017 23 0 0 systemd-journal [314596.065709] [11777] 0 11777 29083 302 12 0 0 net_device_wait [314596.067259] [11778] 0 11778 29083 302 9 0 0 net_device_wait [314596.068796] Out of memory: Kill process 1824 (python) score 0 or sacrifice child [314596.070155] Killed process 1879 (python), UID 89, total-vm:218220kB, anon-rss:9680kB, file-rss:0kB, shmem-rss:0kB [314596.072975] net_device_wait invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [314596.074590] net_device_wait cpuset=/ mems_allowed=0-1 [314596.075764] CPU: 2 PID: 11777 Comm: net_device_wait Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314596.078506] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314596.081554] Call Trace: [314596.082531] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314596.083927] [<ffffffff95b81668>] dump_header+0x90/0x229 [314596.085422] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314596.086924] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314596.088365] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314596.089902] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314596.091470] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314596.093013] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314596.094623] [<ffffffff956193b8>] alloc_pages_current+0x98/0x110 [314596.096139] [<ffffffff955be057>] __page_cache_alloc+0x97/0xb0 [314596.097659] [<ffffffff955c1000>] filemap_fault+0x270/0x420 [314596.099155] [<ffffffffc05fc91e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [314596.100794] [<ffffffffc05fcb1c>] xfs_filemap_fault+0x2c/0x30 [xfs] [314596.102397] [<ffffffff955ee7da>] __do_fault.isra.61+0x8a/0x100 [314596.103939] [<ffffffff955eed8c>] do_read_fault.isra.63+0x4c/0x1b0 [314596.105525] [<ffffffff955f65d0>] handle_mm_fault+0xa20/0xfb0 [314596.107051] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314596.108526] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314596.110098] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314596.111656] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314596.113116] Mem-Info: [314596.114022] active_anon:169710 inactive_anon:14408 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40560 slab_unreclaimable:110563 mapped:853 shmem:172240 pagetables:996 bounce:0 free:91337 free_pcp:724 free_cma:0 [314596.122134] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314596.130514] lowmem_reserve[]: 0 2607 24634 24634 [314596.131877] Node 0 DMA32 free:101608kB min:13744kB low:17180kB high:20616kB active_anon:88kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26252kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:1180kB local_pcp:716kB free_cma:0kB writeback_tmp:0kB pages_scanned:42 all_unreclaimable? yes [314596.140888] lowmem_reserve[]: 0 0 22026 22026 [314596.142253] Node 0 Normal free:115928kB min:116096kB low:145120kB high:174144kB active_anon:39028kB inactive_anon:26244kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:36kB shmem:41272kB slab_reclaimable:81576kB slab_unreclaimable:198712kB kernel_stack:1776kB pagetables:1632kB unstable:0kB bounce:0kB free_pcp:472kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314596.152517] lowmem_reserve[]: 0 0 0 0 [314596.153809] Node 1 Normal free:132000kB min:132216kB low:165268kB high:198324kB active_anon:639724kB inactive_anon:31388kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:3872kB shmem:647688kB slab_reclaimable:74180kB slab_unreclaimable:217256kB kernel_stack:2656kB pagetables:2352kB unstable:0kB bounce:0kB free_pcp:1244kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314596.164180] lowmem_reserve[]: 0 0 0 0 [314596.165489] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314596.169214] Node 0 DMA32: 27*4kB (EM) 26*8kB (UEM) 5*16kB (UM) 6*32kB (UEM) 8*64kB (UM) 15*128kB (UEM) 7*256kB (EM) 4*512kB (M) 4*1024kB (EM) 4*2048kB (UM) 20*4096kB (M) = 101068kB [314596.173518] Node 0 Normal: 2292*4kB (EM) 1720*8kB (UEM) 1119*16kB (UEM) 693*32kB (UEM) 407*64kB (UEM) 184*128kB (UEM) 13*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115936kB [314596.177773] Node 1 Normal: 1218*4kB (UEM) 2079*8kB (UEM) 1429*16kB (UEM) 1082*32kB (UEM) 750*64kB (UM) 12*128kB (M) 1*256kB (U) 1*512kB (U) 3*1024kB (U) 0*2048kB 0*4096kB = 132368kB [314596.182169] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314596.184315] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314596.186405] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314596.188526] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314596.190607] 172284 total pagecache pages [314596.191989] 0 pages in swap cache [314596.193269] Swap cache stats: add 0, delete 0, find 0/0 [314596.194857] Free swap = 0kB [314596.196077] Total swap = 0kB [314596.197297] 13055868 pages RAM [314596.198537] 0 pages HighMem/MovableOnly [314596.199919] 323772 pages reserved [314596.201194] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314596.203204] [ 618] 0 618 49643 96 30 0 0 lvmetad [314596.205239] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314596.207366] [ 801] 0 801 13883 110 29 0 -1000 auditd [314596.209385] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314596.211423] [ 861] 0 861 13212 203 30 0 0 smartd [314596.213418] [ 872] 0 872 11042 91 29 0 0 qemu-ga [314596.215412] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314596.217448] [ 898] 38 898 7457 127 19 0 0 ntpd [314596.219377] [ 916] 0 916 6596 77 17 0 0 systemd-logind [314596.221427] [ 1486] 0 1486 25746 531 49 0 0 dhclient [314596.223384] [ 1819] 997 1819 216185 1865 51 0 0 lustre_exporter [314596.225424] [ 1820] 997 1820 238088 1566 58 0 0 pushprox-client [314596.227451] [ 1823] 99 1823 14304 369 30 0 0 dnsmasq [314596.229349] [ 1824] 89 1824 56604 2425 62 0 0 python [314596.231226] [ 1949] 0 1949 22964 264 45 0 0 master [314596.233096] [ 1951] 89 1951 23007 255 48 0 0 qmgr [314596.234914] [ 2202] 0 2202 28244 258 57 0 -1000 sshd [314596.236721] [ 2207] 0 2207 63618 394 31 0 0 rsyslogd [314596.238565] [ 2208] 0 2208 27552 31 11 0 0 agetty [314596.240368] [ 2209] 0 2209 6477 50 18 0 0 atd [314596.242129] [ 2210] 0 2210 27552 31 8 0 0 agetty [314596.243920] [ 2214] 0 2214 31597 155 18 0 0 crond [314596.245669] [16982] 0 16982 29083 302 14 0 0 net_device_wait [314596.247555] [16983] 0 16983 29083 302 12 0 0 net_device_wait [314596.249426] [21402] 0 21402 38696 325 77 0 0 sshd [314596.251123] [21404] 0 21404 28953 165 11 0 0 bash [314596.252823] [27616] 89 27616 22990 257 46 0 0 pickup [314596.254534] [11762] 0 11762 9851 1017 23 0 0 systemd-journal [314596.256354] [11777] 0 11777 29083 302 12 0 0 net_device_wait [314596.258155] [11778] 0 11778 29083 302 9 0 0 net_device_wait [314596.259945] Out of memory: Kill process 1824 (python) score 0 or sacrifice child [314596.261510] Killed process 1824 (python), UID 89, total-vm:226416kB, anon-rss:9700kB, file-rss:0kB, shmem-rss:0kB [314596.364126] systemd-journal invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [314596.365556] systemd-journal cpuset=/ mems_allowed=0-1 [314596.366555] CPU: 1 PID: 11762 Comm: systemd-journal Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314596.368754] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314596.370964] Call Trace: [314596.371680] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314596.372716] [<ffffffff95b81668>] dump_header+0x90/0x229 [314596.373766] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314596.374848] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314596.375931] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314596.377039] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314596.378190] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314596.379271] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314596.380443] [<ffffffff956193b8>] alloc_pages_current+0x98/0x110 [314596.381568] [<ffffffff955be057>] __page_cache_alloc+0x97/0xb0 [314596.382670] [<ffffffff955c1000>] filemap_fault+0x270/0x420 [314596.383783] [<ffffffffc05fc91e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [314596.384980] [<ffffffffc05fcb1c>] xfs_filemap_fault+0x2c/0x30 [xfs] [314596.386136] [<ffffffff955ee7da>] __do_fault.isra.61+0x8a/0x100 [314596.387257] [<ffffffff955eed8c>] do_read_fault.isra.63+0x4c/0x1b0 [314596.388413] [<ffffffff955f65d0>] handle_mm_fault+0xa20/0xfb0 [314596.389515] [<ffffffff95794326>] ? sprintf+0x56/0x80 [314596.390535] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314596.391635] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314596.392771] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314596.393894] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314596.394989] Mem-Info: [314596.395675] active_anon:167301 inactive_anon:14408 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40560 slab_unreclaimable:110563 mapped:744 shmem:172240 pagetables:996 bounce:0 free:91332 free_pcp:1218 free_cma:0 [314596.401968] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314596.408601] lowmem_reserve[]: 0 2607 24634 24634 [314596.409760] Node 0 DMA32 free:101608kB min:13744kB low:17180kB high:20616kB active_anon:88kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26252kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:1176kB local_pcp:144kB free_cma:0kB writeback_tmp:0kB pages_scanned:39 all_unreclaimable? yes [314596.416926] lowmem_reserve[]: 0 0 22026 22026 [314596.418111] Node 0 Normal free:115908kB min:116096kB low:145120kB high:174144kB active_anon:29392kB inactive_anon:26244kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:41272kB slab_reclaimable:81576kB slab_unreclaimable:198712kB kernel_stack:1776kB pagetables:1632kB unstable:0kB bounce:0kB free_pcp:2112kB local_pcp:188kB free_cma:0kB writeback_tmp:0kB pages_scanned:24 all_unreclaimable? yes [314596.426304] lowmem_reserve[]: 0 0 0 0 [314596.427419] Node 1 Normal free:132000kB min:132216kB low:165268kB high:198324kB active_anon:639724kB inactive_anon:31388kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:3872kB shmem:647688kB slab_reclaimable:74180kB slab_unreclaimable:217256kB kernel_stack:2656kB pagetables:2352kB unstable:0kB bounce:0kB free_pcp:1584kB local_pcp:388kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314596.435714] lowmem_reserve[]: 0 0 0 0 [314596.436884] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314596.440044] Node 0 DMA32: 27*4kB (EM) 26*8kB (UEM) 5*16kB (UM) 6*32kB (UEM) 8*64kB (UM) 15*128kB (UEM) 7*256kB (EM) 4*512kB (M) 4*1024kB (EM) 4*2048kB (UM) 20*4096kB (M) = 101068kB [314596.443666] Node 0 Normal: 1855*4kB (UEM) 1701*8kB (UEM) 1085*16kB (UEM) 697*32kB (UEM) 407*64kB (UEM) 184*128kB (UEM) 21*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115668kB [314596.447297] Node 1 Normal: 1201*4kB (UEM) 2080*8kB (UEM) 1428*16kB (EM) 1081*32kB (EM) 750*64kB (UM) 12*128kB (M) 1*256kB (U) 1*512kB (U) 3*1024kB (U) 0*2048kB 0*4096kB = 132260kB [314596.451006] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314596.452787] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314596.454519] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314596.456283] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314596.458021] 172066 total pagecache pages [314596.459231] 0 pages in swap cache [314596.460372] Swap cache stats: add 0, delete 0, find 0/0 [314596.461750] Free swap = 0kB [314596.462845] Total swap = 0kB [314596.463943] 13055868 pages RAM [314596.465054] 0 pages HighMem/MovableOnly [314596.466253] 323772 pages reserved [314596.467385] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314596.469063] [ 618] 0 618 49643 96 30 0 0 lvmetad [314596.470761] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314596.472510] [ 801] 0 801 13883 110 29 0 -1000 auditd [314596.474196] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314596.475905] [ 861] 0 861 13212 203 30 0 0 smartd [314596.477562] [ 872] 0 872 11042 91 29 0 0 qemu-ga [314596.479215] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314596.480901] [ 898] 38 898 7457 127 19 0 0 ntpd [314596.482499] [ 916] 0 916 6596 77 17 0 0 systemd-logind [314596.484183] [ 1486] 0 1486 25746 531 49 0 0 dhclient [314596.485800] [ 1819] 997 1819 216185 1865 51 0 0 lustre_exporter [314596.487474] [ 1820] 997 1820 238088 1566 58 0 0 pushprox-client [314596.489136] [ 1823] 99 1823 14304 369 30 0 0 dnsmasq [314596.490713] [ 1949] 0 1949 22964 264 45 0 0 master [314596.492247] [ 1951] 89 1951 23007 255 48 0 0 qmgr [314596.493763] [ 2202] 0 2202 28244 258 57 0 -1000 sshd [314596.495253] [ 2207] 0 2207 63618 394 31 0 0 rsyslogd [314596.496817] [ 2208] 0 2208 27552 31 11 0 0 agetty [314596.498308] [ 2209] 0 2209 6477 50 18 0 0 atd [314596.499754] [ 2210] 0 2210 27552 31 8 0 0 agetty [314596.501211] [ 2214] 0 2214 31597 155 18 0 0 crond [314596.502652] [16982] 0 16982 29083 302 14 0 0 net_device_wait [314596.504170] [16983] 0 16983 29083 302 12 0 0 net_device_wait [314596.505677] [21402] 0 21402 38696 325 77 0 0 sshd [314596.507074] [21404] 0 21404 28953 165 11 0 0 bash [314596.508436] [27616] 89 27616 22990 257 46 0 0 pickup [314596.509829] [11762] 0 11762 9851 1027 23 0 0 systemd-journal [314596.511288] [11777] 0 11777 2669 24 10 0 0 systemd-notify [314596.512729] [11778] 0 11778 2669 25 10 0 0 systemd-notify [314596.514141] [11779] 0 11779 2146 22 8 0 0 systemd-cgroups [314596.515559] Out of memory: Kill process 1819 (lustre_exporter) score 0 or sacrifice child [314596.516961] Killed process 1819 (lustre_exporter), UID 997, total-vm:864740kB, anon-rss:7460kB, file-rss:0kB, shmem-rss:0kB [314596.518895] systemd-notify invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [314596.525822] systemd-notify cpuset=/ mems_allowed=0-1 [314596.529328] CPU: 8 PID: 11777 Comm: systemd-notify Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314596.539926] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314596.553150] Call Trace: [314596.556214] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314596.560956] [<ffffffff95b81668>] dump_header+0x90/0x229 [314596.565196] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314596.567992] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314596.571406] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314596.576923] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314596.594864] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314596.598900] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314596.606165] [<ffffffff956193b8>] alloc_pages_current+0x98/0x110 [314596.607727] [<ffffffff955be057>] __page_cache_alloc+0x97/0xb0 [314596.609542] [<ffffffff955c1000>] filemap_fault+0x270/0x420 [314596.612483] [<ffffffffc05fc91e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [314596.616701] [<ffffffffc05fcb1c>] xfs_filemap_fault+0x2c/0x30 [xfs] [314596.619571] [<ffffffff955ee7da>] __do_fault.isra.61+0x8a/0x100 [314596.622683] [<ffffffff955eed8c>] do_read_fault.isra.63+0x4c/0x1b0 [314596.626049] [<ffffffff955f65d0>] handle_mm_fault+0xa20/0xfb0 [314596.629536] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314596.632334] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314596.635095] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314596.637810] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314596.642048] Mem-Info: [314596.643426] active_anon:165366 inactive_anon:14408 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40560 slab_unreclaimable:110563 mapped:744 shmem:172240 pagetables:996 bounce:0 free:91894 free_pcp:680 free_cma:0 [314596.664426] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314596.689251] lowmem_reserve[]: 0 2607 24634 24634 [314596.693173] Node 0 DMA32 free:101908kB min:13744kB low:17180kB high:20616kB active_anon:88kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26252kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:168kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4 all_unreclaimable? yes [314596.721965] lowmem_reserve[]: 0 0 22026 22026 [314596.725905] Node 0 Normal free:116072kB min:116096kB low:145120kB high:174144kB active_anon:24572kB inactive_anon:26244kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:41272kB slab_reclaimable:81576kB slab_unreclaimable:198712kB kernel_stack:1776kB pagetables:1632kB unstable:0kB bounce:0kB free_pcp:1348kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:39 all_unreclaimable? yes [314596.759468] lowmem_reserve[]: 0 0 0 0 [314596.763203] Node 1 Normal free:133828kB min:132216kB low:165268kB high:198324kB active_anon:636804kB inactive_anon:31388kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:4308kB shmem:647688kB slab_reclaimable:74180kB slab_unreclaimable:217256kB kernel_stack:2656kB pagetables:2352kB unstable:0kB bounce:0kB free_pcp:1052kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:28 all_unreclaimable? yes [314596.786464] lowmem_reserve[]: 0 0 0 0 [314596.788856] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314596.796319] Node 0 DMA32: 23*4kB (UE) 39*8kB (UEM) 19*16kB (UM) 20*32kB (UEM) 11*64kB (UM) 13*128kB (UEM) 7*256kB (UEM) 4*512kB (UM) 4*1024kB (UEM) 2*2048kB (M) 21*4096kB (M) = 101764kB [314596.805576] Node 0 Normal: 1814*4kB (UEM) 1651*8kB (UEM) 1079*16kB (UEM) 696*32kB (UEM) 412*64kB (UEM) 199*128kB (UEM) 20*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 116960kB [314596.812378] Node 1 Normal: 1200*4kB (EM) 2053*8kB (EM) 1429*16kB (EM) 1089*32kB (UEM) 756*64kB (UM) 15*128kB (M) 0*256kB 1*512kB (U) 2*1024kB (U) 0*2048kB 0*4096kB = 131800kB [314596.816233] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314596.818108] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314596.820288] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314596.822054] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314596.823882] 171938 total pagecache pages [314596.825099] 0 pages in swap cache [314596.826294] Swap cache stats: add 0, delete 0, find 0/0 [314596.827850] Free swap = 0kB [314596.829014] Total swap = 0kB [314596.830158] 13055868 pages RAM [314596.831328] 0 pages HighMem/MovableOnly [314596.832524] 323772 pages reserved [314596.833662] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314596.835613] [ 618] 0 618 49643 96 30 0 0 lvmetad [314596.837358] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314596.839556] [ 801] 0 801 13883 110 29 0 -1000 auditd [314596.841346] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314596.843272] [ 861] 0 861 13212 203 30 0 0 smartd [314596.845029] [ 872] 0 872 11042 91 29 0 0 qemu-ga [314596.846801] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314596.848492] [ 898] 38 898 7457 127 19 0 0 ntpd [314596.850097] [ 916] 0 916 6596 77 17 0 0 systemd-logind [314596.851809] [ 1486] 0 1486 25746 531 49 0 0 dhclient [314596.853435] [ 1820] 997 1820 238088 1593 58 0 0 pushprox-client [314596.855120] [ 1823] 99 1823 14304 369 30 0 0 dnsmasq [314596.856700] [ 1949] 0 1949 22964 264 45 0 0 master [314596.858612] [ 1951] 89 1951 23007 255 48 0 0 qmgr [314596.860519] [ 2202] 0 2202 28244 258 57 0 -1000 sshd [314596.864015] [ 2207] 0 2207 63618 394 31 0 0 rsyslogd [314596.867515] [ 2208] 0 2208 27552 31 11 0 0 agetty [314596.869374] [ 2209] 0 2209 6477 50 18 0 0 atd [314596.871155] [ 2210] 0 2210 27552 31 8 0 0 agetty [314596.872959] [ 2214] 0 2214 31597 155 18 0 0 crond [314596.874758] [16982] 0 16982 29083 302 14 0 0 net_device_wait [314596.876669] [16983] 0 16983 29083 309 12 0 0 net_device_wait [314596.878567] [21402] 0 21402 38696 325 77 0 0 sshd [314596.880298] [21404] 0 21404 28953 165 11 0 0 bash [314596.882015] [27616] 89 27616 22990 257 46 0 0 pickup [314596.883751] [11762] 0 11762 9851 1023 23 0 0 systemd-journal [314596.885595] [11777] 0 11777 2669 24 10 0 0 systemd-notify [314596.887417] [11782] 0 11782 27014 21 8 0 0 sleep [314596.889098] Out of memory: Kill process 1820 (pushprox-client) score 0 or sacrifice child [314596.890866] Killed process 1820 (pushprox-client), UID 997, total-vm:952352kB, anon-rss:6372kB, file-rss:0kB, shmem-rss:0kB [314597.166827] kworker/u20:1 invoked oom-killer: gfp_mask=0x80d0, order=0, oom_score_adj=0 [314597.169985] kworker/u20:1 cpuset=/ mems_allowed=0-1 [314597.171098] CPU: 0 PID: 27715 Comm: kworker/u20:1 Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314597.173444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314597.175818] Workqueue: rdma_cm cma_work_handler [rdma_cm] [314597.176981] Call Trace: [314597.177764] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314597.178879] [<ffffffff95b81668>] dump_header+0x90/0x229 [314597.180014] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314597.181193] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314597.182355] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314597.183550] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314597.184843] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314597.186022] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314597.187297] [<ffffffff9543346f>] dma_generic_alloc_coherent+0x8f/0x140 [314597.188589] [<ffffffff9546eb01>] x86_swiotlb_alloc_coherent+0x21/0x50 [314597.189901] [<ffffffffc036f3a9>] mlx5_dma_zalloc_coherent_node+0xb9/0x110 [mlx5_core] [314597.191332] [<ffffffffc036f9ba>] mlx5_frag_buf_alloc_node+0xfa/0x310 [mlx5_core] [314597.192739] [<ffffffffc036f589>] ? mlx5_db_alloc_node+0x69/0x1a0 [mlx5_core] [314597.194066] [<ffffffffc05516d3>] mlx5_ib_create_cq+0x3e3/0x680 [mlx5_ib] [314597.195320] [<ffffffff95628ceb>] ? __kmalloc+0x1eb/0x230 [314597.196431] [<ffffffffc06dc191>] __ib_create_cq+0xa1/0xf0 [ib_core] [314597.197705] [<ffffffffc0936e90>] ? kiblnd_cq_completion+0x160/0x160 [ko2iblnd] [314597.199076] [<ffffffffc092563a>] kiblnd_create_conn+0xcba/0x1a00 [ko2iblnd] [314597.200382] [<ffffffffc093514e>] kiblnd_cm_callback+0x145e/0x23a0 [ko2iblnd] [314597.201689] [<ffffffffc07b9d81>] cma_work_handler+0xa1/0xe0 [rdma_cm] [314597.202944] [<ffffffff954bdfbf>] process_one_work+0x17f/0x440 [314597.204102] [<ffffffff954bf0d6>] worker_thread+0x126/0x3c0 [314597.205233] [<ffffffff954befb0>] ? manage_workers.isra.26+0x2a0/0x2a0 [314597.206481] [<ffffffff954c5f91>] kthread+0xd1/0xe0 [314597.207551] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314597.208746] [<ffffffff95b99ddd>] ret_from_fork_nospec_begin+0x7/0x21 [314597.209988] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314597.211181] Mem-Info: [314597.211947] active_anon:163144 inactive_anon:14404 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40565 slab_unreclaimable:110601 mapped:886 shmem:172236 pagetables:821 bounce:0 free:91420 free_pcp:237 free_cma:0 [314597.218615] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314597.225518] lowmem_reserve[]: 0 2607 24634 24634 [314597.226690] Node 0 DMA32 free:101680kB min:13744kB low:17180kB high:20616kB active_anon:136kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314597.233863] lowmem_reserve[]: 0 0 22026 22026 [314597.235025] Node 0 Normal free:115988kB min:116096kB low:145120kB high:174144kB active_anon:22444kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:41260kB slab_reclaimable:81592kB slab_unreclaimable:198776kB kernel_stack:1776kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314597.243220] lowmem_reserve[]: 0 0 0 0 [314597.244364] Node 1 Normal free:132200kB min:132216kB low:165268kB high:198324kB active_anon:629996kB inactive_anon:31384kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:3916kB shmem:647684kB slab_reclaimable:74184kB slab_unreclaimable:217340kB kernel_stack:2640kB pagetables:2148kB unstable:0kB bounce:0kB free_pcp:828kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:2810 all_unreclaimable? yes [314597.252930] lowmem_reserve[]: 0 0 0 0 [314597.254123] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314597.257380] Node 0 DMA32: 37*4kB (UEM) 38*8kB (EM) 16*16kB (UM) 16*32kB (UEM) 6*64kB (M) 10*128kB (UEM) 6*256kB (UEM) 6*512kB (UM) 4*1024kB (UEM) 2*2048kB (M) 21*4096kB (M) = 101700kB [314597.261314] Node 0 Normal: 1877*4kB (EM) 1653*8kB (EM) 1064*16kB (UEM) 691*32kB (UEM) 411*64kB (UEM) 195*128kB (EM) 17*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 115996kB [314597.265337] Node 1 Normal: 1351*4kB (UEM) 2081*8kB (EM) 1458*16kB (EM) 1121*32kB (UEM) 773*64kB (UM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 132644kB [314597.269591] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314597.271763] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314597.273888] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314597.276046] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314597.278163] 172177 total pagecache pages [314597.279560] 0 pages in swap cache [314597.280864] Swap cache stats: add 0, delete 0, find 0/0 [314597.282461] Free swap = 0kB [314597.283701] Total swap = 0kB [314597.284939] 13055868 pages RAM [314597.286214] 0 pages HighMem/MovableOnly [314597.287587] 323772 pages reserved [314597.288882] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314597.290909] [ 618] 0 618 49643 96 30 0 0 lvmetad [314597.293062] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314597.295199] [ 801] 0 801 13883 110 29 0 -1000 auditd [314597.297252] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314597.299345] [ 861] 0 861 13212 203 30 0 0 smartd [314597.301364] [ 872] 0 872 11042 91 29 0 0 qemu-ga [314597.303377] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314597.305431] [ 898] 38 898 7457 127 19 0 0 ntpd [314597.307374] [ 916] 0 916 6596 77 17 0 0 systemd-logind [314597.309450] [ 1486] 0 1486 25746 531 49 0 0 dhclient [314597.311429] [ 1823] 99 1823 14304 369 30 0 0 dnsmasq [314597.313386] [ 1949] 0 1949 22964 264 45 0 0 master [314597.315307] [ 1951] 89 1951 23007 255 48 0 0 qmgr [314597.317193] [ 2202] 0 2202 28244 258 57 0 -1000 sshd [314597.319064] [ 2207] 0 2207 63618 394 31 0 0 rsyslogd [314597.320975] [ 2208] 0 2208 27552 31 11 0 0 agetty [314597.322841] [ 2209] 0 2209 6477 50 18 0 0 atd [314597.324641] [ 2210] 0 2210 27552 31 8 0 0 agetty [314597.326474] [ 2214] 0 2214 31597 155 18 0 0 crond [314597.328274] [16982] 0 16982 29083 302 14 0 0 net_device_wait [314597.330210] [16983] 0 16983 29083 302 12 0 0 net_device_wait [314597.332137] [21402] 0 21402 38696 325 77 0 0 sshd [314597.333897] [21404] 0 21404 28953 165 11 0 0 bash [314597.335639] [27616] 89 27616 22990 257 46 0 0 pickup [314597.337399] [11762] 0 11762 9851 1020 23 0 0 systemd-journal [314597.339260] [11777] 0 11777 2669 24 10 0 0 systemd-notify [314597.341113] [11782] 0 11782 27014 22 8 0 0 sleep [314597.342829] [11783] 0 11783 2146 23 8 0 0 systemd-cgroups [314597.344656] Out of memory: Kill process 11762 (systemd-journal) score 0 or sacrifice child [314597.346405] Killed process 11762 (systemd-journal), UID 0, total-vm:39404kB, anon-rss:296kB, file-rss:0kB, shmem-rss:3784kB [314597.349142] systemd-notify invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [314597.350546] systemd-notify cpuset=/ mems_allowed=0-1 [314597.351565] CPU: 3 PID: 11777 Comm: systemd-notify Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314597.353781] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314597.356537] Call Trace: [314597.357271] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314597.358320] [<ffffffff95b81668>] dump_header+0x90/0x229 [314597.359373] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314597.360481] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314597.361607] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314597.362868] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314597.364131] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314597.365240] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314597.366463] [<ffffffff956193b8>] alloc_pages_current+0x98/0x110 [314597.367603] [<ffffffff955be057>] __page_cache_alloc+0x97/0xb0 [314597.368726] [<ffffffff955c1000>] filemap_fault+0x270/0x420 [314597.369845] [<ffffffffc05fc91e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [314597.371073] [<ffffffffc05fcb1c>] xfs_filemap_fault+0x2c/0x30 [xfs] [314597.372250] [<ffffffff955ee7da>] __do_fault.isra.61+0x8a/0x100 [314597.373382] [<ffffffff955eed8c>] do_read_fault.isra.63+0x4c/0x1b0 [314597.374556] [<ffffffff955f65d0>] handle_mm_fault+0xa20/0xfb0 [314597.375665] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314597.376801] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314597.377934] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314597.379076] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314597.380179] Mem-Info: [314597.380899] active_anon:163144 inactive_anon:14404 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40565 slab_unreclaimable:110601 mapped:14 shmem:172236 pagetables:821 bounce:0 free:91419 free_pcp:430 free_cma:0 [314597.387251] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314597.393961] lowmem_reserve[]: 0 2607 24634 24634 [314597.395117] Node 0 DMA32 free:101680kB min:13744kB low:17180kB high:20616kB active_anon:136kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314597.402325] lowmem_reserve[]: 0 0 22026 22026 [314597.403514] Node 0 Normal free:115988kB min:116096kB low:145120kB high:174144kB active_anon:22444kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:41260kB slab_reclaimable:81592kB slab_unreclaimable:198776kB kernel_stack:1776kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:148kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314597.411789] lowmem_reserve[]: 0 0 0 0 [314597.412921] Node 1 Normal free:132196kB min:132216kB low:165268kB high:198324kB active_anon:629996kB inactive_anon:31384kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:428kB shmem:647684kB slab_reclaimable:74184kB slab_unreclaimable:217340kB kernel_stack:2640kB pagetables:2148kB unstable:0kB bounce:0kB free_pcp:1572kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:86 all_unreclaimable? yes [314597.421303] lowmem_reserve[]: 0 0 0 0 [314597.422462] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314597.425630] Node 0 DMA32: 37*4kB (UEM) 38*8kB (EM) 16*16kB (UM) 16*32kB (UEM) 6*64kB (M) 10*128kB (UEM) 6*256kB (UEM) 6*512kB (UM) 4*1024kB (UEM) 2*2048kB (M) 21*4096kB (M) = 101700kB [314597.429331] Node 0 Normal: 1887*4kB (UEM) 1658*8kB (UEM) 1063*16kB (UEM) 691*32kB (UEM) 411*64kB (UEM) 195*128kB (UEM) 17*256kB (UM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 116060kB [314597.433030] Node 1 Normal: 1292*4kB (UEM) 2073*8kB (UEM) 1459*16kB (UEM) 1122*32kB (UEM) 770*64kB (UM) 13*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 131944kB [314597.436610] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314597.438398] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314597.440149] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314597.441926] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314597.443677] 172177 total pagecache pages [314597.444897] 0 pages in swap cache [314597.446046] Swap cache stats: add 0, delete 0, find 0/0 [314597.447429] Free swap = 0kB [314597.448529] Total swap = 0kB [314597.449631] 13055868 pages RAM [314597.450771] 0 pages HighMem/MovableOnly [314597.451977] 323772 pages reserved [314597.453116] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314597.454809] [ 618] 0 618 49643 96 30 0 0 lvmetad [314597.456513] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314597.458298] [ 801] 0 801 13883 110 29 0 -1000 auditd [314597.460000] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314597.461721] [ 861] 0 861 13212 203 30 0 0 smartd [314597.463395] [ 872] 0 872 11042 91 29 0 0 qemu-ga [314597.465071] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314597.466778] [ 898] 38 898 7457 127 19 0 0 ntpd [314597.468748] [ 916] 0 916 6596 77 17 0 0 systemd-logind [314597.470449] [ 1486] 0 1486 25746 531 49 0 0 dhclient [314597.472092] [ 1823] 99 1823 14304 369 30 0 0 dnsmasq [314597.473701] [ 1949] 0 1949 22964 264 45 0 0 master [314597.475293] [ 1951] 89 1951 23007 255 48 0 0 qmgr [314597.476842] [ 2202] 0 2202 28244 258 57 0 -1000 sshd [314597.478388] [ 2207] 0 2207 63618 394 31 0 0 rsyslogd [314597.479954] [ 2208] 0 2208 27552 31 11 0 0 agetty [314597.481489] [ 2209] 0 2209 6477 50 18 0 0 atd [314597.482970] [ 2210] 0 2210 27552 31 8 0 0 agetty [314597.484461] [ 2214] 0 2214 31597 155 18 0 0 crond [314597.485934] [16982] 0 16982 29083 302 14 0 0 net_device_wait [314597.487501] [16983] 0 16983 29083 302 12 0 0 net_device_wait [314597.489053] [21402] 0 21402 38696 325 77 0 0 sshd [314597.490478] [21404] 0 21404 28953 165 11 0 0 bash [314597.491908] [27616] 89 27616 22990 257 46 0 0 pickup [314597.493326] [11777] 0 11777 2669 24 10 0 0 systemd-notify [314597.494808] [11782] 0 11782 27014 22 8 0 0 sleep [314597.496220] [11783] 0 11783 2146 23 8 0 0 systemd-cgroups [314597.497739] Out of memory: Kill process 1486 (dhclient) score 0 or sacrifice child [314597.499103] Killed process 1486 (dhclient), UID 0, total-vm:102984kB, anon-rss:2124kB, file-rss:0kB, shmem-rss:0kB [314597.511859] auditd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=-1000 [314597.513239] auditd cpuset=/ mems_allowed=0-1 [314597.514175] CPU: 2 PID: 801 Comm: auditd Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314597.516288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314597.518481] Call Trace: [314597.519203] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314597.520239] [<ffffffff95b81668>] dump_header+0x90/0x229 [314597.521298] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314597.522388] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314597.523472] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314597.524579] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314597.525719] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314597.526805] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314597.527980] [<ffffffff956193b8>] alloc_pages_current+0x98/0x110 [314597.529116] [<ffffffff955be057>] __page_cache_alloc+0x97/0xb0 [314597.530231] [<ffffffff955c1000>] filemap_fault+0x270/0x420 [314597.531337] [<ffffffffc05fc91e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [314597.532537] [<ffffffffc05fcb1c>] xfs_filemap_fault+0x2c/0x30 [xfs] [314597.533699] [<ffffffff955ee7da>] __do_fault.isra.61+0x8a/0x100 [314597.534818] [<ffffffff955eed8c>] do_read_fault.isra.63+0x4c/0x1b0 [314597.535964] [<ffffffff955f65d0>] handle_mm_fault+0xa20/0xfb0 [314597.537068] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314597.538169] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314597.539299] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314597.540414] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314597.541503] Mem-Info: [314597.542204] active_anon:162633 inactive_anon:14404 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40565 slab_unreclaimable:110601 mapped:0 shmem:172236 pagetables:821 bounce:0 free:91417 free_pcp:771 free_cma:0 [314597.548487] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314597.555121] lowmem_reserve[]: 0 2607 24634 24634 [314597.556259] Node 0 DMA32 free:101680kB min:13744kB low:17180kB high:20616kB active_anon:136kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314597.563734] lowmem_reserve[]: 0 0 22026 22026 [314597.564924] Node 0 Normal free:115988kB min:116096kB low:145120kB high:174144kB active_anon:22444kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:41260kB slab_reclaimable:81592kB slab_unreclaimable:198776kB kernel_stack:1776kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:328kB local_pcp:20kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314597.573138] lowmem_reserve[]: 0 0 0 0 [314597.574286] Node 1 Normal free:132064kB min:132216kB low:165268kB high:198324kB active_anon:627952kB inactive_anon:31384kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:647684kB slab_reclaimable:74184kB slab_unreclaimable:217340kB kernel_stack:2640kB pagetables:2148kB unstable:0kB bounce:0kB free_pcp:2348kB local_pcp:316kB free_cma:0kB writeback_tmp:0kB pages_scanned:28 all_unreclaimable? yes [314597.582580] lowmem_reserve[]: 0 0 0 0 [314597.583738] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314597.586926] Node 0 DMA32: 37*4kB (UEM) 38*8kB (EM) 16*16kB (UM) 16*32kB (UEM) 6*64kB (M) 10*128kB (UEM) 6*256kB (UEM) 6*512kB (UM) 4*1024kB (UEM) 2*2048kB (M) 21*4096kB (M) = 101700kB [314597.590620] Node 0 Normal: 1933*4kB (UEM) 1656*8kB (UEM) 1059*16kB (UEM) 691*32kB (UEM) 411*64kB (UEM) 195*128kB (UEM) 17*256kB (UM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 116164kB [314597.594286] Node 1 Normal: 1173*4kB (UEM) 2070*8kB (EM) 1451*16kB (EM) 1120*32kB (UEM) 766*64kB (UM) 16*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 131380kB [314597.597830] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314597.599602] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314597.601342] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314597.603113] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314597.604840] 171959 total pagecache pages [314597.606055] 0 pages in swap cache [314597.607205] Swap cache stats: add 0, delete 0, find 0/0 [314597.608567] Free swap = 0kB [314597.609657] Total swap = 0kB [314597.610751] 13055868 pages RAM [314597.611862] 0 pages HighMem/MovableOnly [314597.613062] 323772 pages reserved [314597.614196] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314597.615865] [ 618] 0 618 49643 96 30 0 0 lvmetad [314597.617559] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314597.619314] [ 801] 0 801 13883 110 29 0 -1000 auditd [314597.620994] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314597.622692] [ 861] 0 861 13212 203 30 0 0 smartd [314597.624360] [ 872] 0 872 11042 91 29 0 0 qemu-ga [314597.626023] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314597.627703] [ 898] 38 898 7457 127 19 0 0 ntpd [314597.629305] [ 916] 0 916 6596 77 17 0 0 systemd-logind [314597.631001] [ 1823] 99 1823 14304 369 30 0 0 dnsmasq [314597.632614] [ 1949] 0 1949 22964 264 45 0 0 master [314597.634198] [ 1951] 89 1951 23007 255 48 0 0 qmgr [314597.635751] [ 2202] 0 2202 28244 258 57 0 -1000 sshd [314597.637283] [ 2207] 0 2207 63618 394 31 0 0 rsyslogd [314597.638852] [ 2208] 0 2208 27552 31 11 0 0 agetty [314597.640380] [ 2209] 0 2209 6477 50 18 0 0 atd [314597.641865] [ 2210] 0 2210 27552 31 8 0 0 agetty [314597.643368] [ 2214] 0 2214 31597 155 18 0 0 crond [314597.644841] [16982] 0 16982 29083 302 14 0 0 net_device_wait [314597.646392] [16983] 0 16983 29083 302 12 0 0 net_device_wait [314597.647951] [21402] 0 21402 38696 325 77 0 0 sshd [314597.649382] [21404] 0 21404 28953 165 11 0 0 bash [314597.650802] [27616] 89 27616 22990 257 46 0 0 pickup [314597.652216] [11777] 0 11777 2669 24 10 0 0 systemd-notify [314597.653696] [11782] 0 11782 27014 22 8 0 0 sleep [314597.655073] [11783] 0 11783 2146 23 8 0 0 systemd-cgroups [314597.656556] [11784] 0 11784 82 1 5 0 0 systemd-cgroups [314597.658022] [11785] 0 11785 79 1 3 0 0 systemd-cgroups [314597.659470] Out of memory: Kill process 2207 (rsyslogd) score 0 or sacrifice child [314597.660786] Killed process 2207 (rsyslogd), UID 0, total-vm:254472kB, anon-rss:1576kB, file-rss:0kB, shmem-rss:0kB [314597.664034] kworker/u20:1 invoked oom-killer: gfp_mask=0x80d0, order=0, oom_score_adj=0 [314597.666202] kworker/u20:1 cpuset=/ mems_allowed=0-1 [314597.667356] CPU: 8 PID: 27715 Comm: kworker/u20:1 Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314597.670106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314597.672968] Workqueue: rdma_cm cma_work_handler [rdma_cm] [314597.674353] Call Trace: [314597.675243] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314597.676578] [<ffffffff95b81668>] dump_header+0x90/0x229 [314597.677940] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314597.679341] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314597.680726] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314597.682173] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314597.683664] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314597.685067] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314597.686598] [<ffffffff9543346f>] dma_generic_alloc_coherent+0x8f/0x140 [314597.688179] [<ffffffff9546eb01>] x86_swiotlb_alloc_coherent+0x21/0x50 [314597.689767] [<ffffffffc036f3a9>] mlx5_dma_zalloc_coherent_node+0xb9/0x110 [mlx5_core] [314597.691563] [<ffffffffc036f9ba>] mlx5_frag_buf_alloc_node+0xfa/0x310 [mlx5_core] [314597.693283] [<ffffffffc0585e8e>] _create_kernel_qp.isra.66+0x41c/0x80e [mlx5_ib] [314597.695002] [<ffffffff956eb1a3>] ? __create_file+0xf3/0x2c0 [314597.696426] [<ffffffffc05863e0>] create_kernel_qp.isra.76+0x160/0x52a [mlx5_ib] [314597.698143] [<ffffffffc056d2a5>] mlx5_ib_create_qp+0x13c5/0x1570 [mlx5_ib] [314597.699797] [<ffffffffc032e3c2>] ? backport___xa_alloc+0xc2/0x190 [mlx_compat] [314597.701525] [<ffffffffc06dbcd6>] ib_create_qp+0xa6/0x3f0 [ib_core] [314597.703081] [<ffffffffc07b62f1>] rdma_create_qp+0x41/0xc0 [rdma_cm] [314597.704630] [<ffffffffc092581c>] kiblnd_create_conn+0xe9c/0x1a00 [ko2iblnd] [314597.706297] [<ffffffffc093514e>] kiblnd_cm_callback+0x145e/0x23a0 [ko2iblnd] [314597.707980] [<ffffffffc07b9d81>] cma_work_handler+0xa1/0xe0 [rdma_cm] [314597.709576] [<ffffffff954bdfbf>] process_one_work+0x17f/0x440 [314597.711057] [<ffffffff954bf0d6>] worker_thread+0x126/0x3c0 [314597.712482] [<ffffffff954befb0>] ? manage_workers.isra.26+0x2a0/0x2a0 [314597.714075] [<ffffffff954c5f91>] kthread+0xd1/0xe0 [314597.715393] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314597.716938] [<ffffffff95b99ddd>] ret_from_fork_nospec_begin+0x7/0x21 [314597.718514] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314597.720037] Mem-Info: [314597.720940] active_anon:162268 inactive_anon:14404 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40565 slab_unreclaimable:110601 mapped:0 shmem:172236 pagetables:821 bounce:0 free:91406 free_pcp:901 free_cma:0 [314597.729441] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314597.738433] lowmem_reserve[]: 0 2607 24634 24634 [314597.739879] Node 0 DMA32 free:101680kB min:13744kB low:17180kB high:20616kB active_anon:136kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314597.749261] lowmem_reserve[]: 0 0 22026 22026 [314597.750656] Node 0 Normal free:115988kB min:116096kB low:145120kB high:174144kB active_anon:22152kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:41260kB slab_reclaimable:81592kB slab_unreclaimable:198776kB kernel_stack:1776kB pagetables:1136kB unstable:0kB bounce:0kB free_pcp:716kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314597.761196] lowmem_reserve[]: 0 0 0 0 [314597.762547] Node 1 Normal free:132144kB min:132216kB low:165268kB high:198324kB active_anon:626784kB inactive_anon:31384kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:647684kB slab_reclaimable:74184kB slab_unreclaimable:217340kB kernel_stack:2640kB pagetables:2148kB unstable:0kB bounce:0kB free_pcp:2888kB local_pcp:240kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314597.773489] lowmem_reserve[]: 0 0 0 0 [314597.774921] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314597.778919] Node 0 DMA32: 37*4kB (UEM) 38*8kB (EM) 16*16kB (UM) 16*32kB (UEM) 6*64kB (M) 10*128kB (UEM) 6*256kB (UEM) 6*512kB (UM) 4*1024kB (UEM) 2*2048kB (M) 21*4096kB (M) = 101700kB [314597.783580] Node 0 Normal: 1933*4kB (UEM) 1656*8kB (UEM) 1059*16kB (UEM) 691*32kB (UEM) 411*64kB (UEM) 195*128kB (UEM) 17*256kB (UM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 116164kB [314597.788217] Node 1 Normal: 1268*4kB (UEM) 2068*8kB (UEM) 1450*16kB (UEM) 1117*32kB (UEM) 769*64kB (UM) 15*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 131952kB [314597.792772] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314597.795035] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314597.797253] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314597.799509] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314597.801708] 172177 total pagecache pages [314597.805197] 0 pages in swap cache [314597.806584] Swap cache stats: add 0, delete 0, find 0/0 [314597.808280] Free swap = 0kB [314597.809575] Total swap = 0kB [314597.810881] 13055868 pages RAM [314597.812198] 0 pages HighMem/MovableOnly [314597.813642] 323772 pages reserved [314597.814997] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314597.817103] [ 618] 0 618 49643 96 30 0 0 lvmetad [314597.819254] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314597.821500] [ 801] 0 801 13883 110 29 0 -1000 auditd [314597.823638] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314597.825809] [ 861] 0 861 13212 203 30 0 0 smartd [314597.827920] [ 872] 0 872 11042 91 29 0 0 qemu-ga [314597.830025] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314597.832186] [ 898] 38 898 7457 127 19 0 0 ntpd [314597.834224] [ 916] 0 916 6596 77 17 0 0 systemd-logind [314597.836402] [ 1823] 99 1823 14304 369 30 0 0 dnsmasq [314597.838491] [ 1949] 0 1949 22964 264 45 0 0 master [314597.842676] [ 1951] 89 1951 23007 255 48 0 0 qmgr [314597.845204] [ 2202] 0 2202 28244 258 57 0 -1000 sshd [314597.847197] [ 2208] 0 2208 27552 31 11 0 0 agetty [314597.850635] [ 2209] 0 2209 6477 50 18 0 0 atd [314597.852588] [ 2210] 0 2210 27552 31 8 0 0 agetty [314597.854549] [ 2214] 0 2214 31597 155 18 0 0 crond [314597.857127] [16982] 0 16982 29083 302 14 0 0 net_device_wait [314597.861265] [16983] 0 16983 29083 302 12 0 0 net_device_wait [314597.864178] [21402] 0 21402 38696 325 77 0 0 sshd [314597.866314] [21404] 0 21404 28953 165 11 0 0 bash [314597.868649] [27616] 89 27616 22990 257 46 0 0 pickup [314597.871107] [11782] 0 11782 27014 22 8 0 0 sleep [314597.873550] [11785] 0 11785 2168 62 10 0 0 systemd-cgroups [314597.876328] [11786] 0 11786 2168 52 10 0 0 systemd-cgroups [314597.879307] Out of memory: Kill process 1823 (dnsmasq) score 0 or sacrifice child [314597.882028] Killed process 1823 (dnsmasq), UID 99, total-vm:57216kB, anon-rss:1476kB, file-rss:0kB, shmem-rss:0kB [314598.173843] LNet: 17020:0:(o2iblnd_cb.c:3429:kiblnd_check_conns()) Timed out tx for AA.AA.ID26@o2ibxx: 1 seconds [314598.175624] LNet: 17020:0:(o2iblnd_cb.c:3429:kiblnd_check_conns()) Skipped 20 previous similar messages [314599.860187] net_device_wait invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [314599.862995] net_device_wait cpuset=/ mems_allowed=0-1 [314599.864081] CPU: 0 PID: 11787 Comm: net_device_wait Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314599.866367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314599.868611] Call Trace: [314599.869351] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314599.870406] [<ffffffff95b81668>] dump_header+0x90/0x229 [314599.871484] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314599.872595] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314599.873696] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314599.874817] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314599.875987] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314599.877084] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314599.878277] [<ffffffff956193b8>] alloc_pages_current+0x98/0x110 [314599.879427] [<ffffffff955be057>] __page_cache_alloc+0x97/0xb0 [314599.880555] [<ffffffff955c1000>] filemap_fault+0x270/0x420 [314599.881691] [<ffffffffc05fc91e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [314599.882919] [<ffffffffc05fcb1c>] xfs_filemap_fault+0x2c/0x30 [xfs] [314599.884102] [<ffffffff955ee7da>] __do_fault.isra.61+0x8a/0x100 [314599.885238] [<ffffffff955eed8c>] do_read_fault.isra.63+0x4c/0x1b0 [314599.886407] [<ffffffff955f65d0>] handle_mm_fault+0xa20/0xfb0 [314599.887525] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314599.888646] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314599.889787] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314599.891171] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314599.892521] Mem-Info: [314599.893344] active_anon:161796 inactive_anon:14403 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40560 slab_unreclaimable:110713 mapped:11 shmem:172236 pagetables:658 bounce:0 free:91216 free_pcp:353 free_cma:0 [314599.901112] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314599.909548] lowmem_reserve[]: 0 2607 24634 24634 [314599.910917] Node 0 DMA32 free:101700kB min:13744kB low:17180kB high:20616kB active_anon:116kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314599.919959] lowmem_reserve[]: 0 0 22026 22026 [314599.921337] Node 0 Normal free:115748kB min:116096kB low:145120kB high:174144kB active_anon:21788kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:41260kB slab_reclaimable:81572kB slab_unreclaimable:198992kB kernel_stack:1616kB pagetables:816kB unstable:0kB bounce:0kB free_pcp:856kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:29 all_unreclaimable? yes [314599.931455] lowmem_reserve[]: 0 0 0 0 [314599.932744] Node 1 Normal free:131604kB min:132216kB low:165268kB high:198324kB active_anon:625280kB inactive_anon:31380kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:647684kB slab_reclaimable:74184kB slab_unreclaimable:217572kB kernel_stack:2160kB pagetables:1816kB unstable:0kB bounce:0kB free_pcp:556kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? yes [314599.943013] lowmem_reserve[]: 0 0 0 0 [314599.944332] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314599.948075] Node 0 DMA32: 37*4kB (UEM) 38*8kB (EM) 16*16kB (UM) 16*32kB (UEM) 6*64kB (M) 10*128kB (UEM) 6*256kB (UEM) 6*512kB (UM) 4*1024kB (UEM) 2*2048kB (M) 21*4096kB (M) = 101700kB [314599.952451] Node 0 Normal: 1691*4kB (UEM) 1648*8kB (UEM) 1062*16kB (EM) 685*32kB (EM) 411*64kB (UEM) 191*128kB (EM) 21*256kB (UM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 115500kB [314599.956795] Node 1 Normal: 1435*4kB (UEM) 2049*8kB (UEM) 1434*16kB (EM) 1114*32kB (UEM) 763*64kB (UM) 16*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 131604kB [314599.961020] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314599.963165] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314599.965262] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314599.967400] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314599.969484] 172261 total pagecache pages [314599.970878] 0 pages in swap cache [314599.972161] Swap cache stats: add 0, delete 0, find 0/0 [314599.973748] Free swap = 0kB [314599.974976] Total swap = 0kB [314599.976201] 13055868 pages RAM [314599.977443] 0 pages HighMem/MovableOnly [314599.978808] 323772 pages reserved [314599.980088] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314599.982093] [ 618] 0 618 49643 96 30 0 0 lvmetad [314599.984133] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314599.986267] [ 801] 0 801 13883 110 29 0 -1000 auditd [314599.988293] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314599.990359] [ 861] 0 861 13212 203 30 0 0 smartd [314599.992367] [ 872] 0 872 11042 91 29 0 0 qemu-ga [314599.994368] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314599.996410] [ 898] 38 898 7457 127 19 0 0 ntpd [314599.998344] [ 916] 0 916 6596 77 17 0 0 systemd-logind [314600.000423] [ 1949] 0 1949 22964 264 45 0 0 master [314600.002365] [ 1951] 89 1951 23007 255 48 0 0 qmgr [314600.004265] [ 2202] 0 2202 28244 258 57 0 -1000 sshd [314600.006145] [ 2208] 0 2208 27552 31 11 0 0 agetty [314600.008042] [ 2209] 0 2209 6477 50 18 0 0 atd [314600.009890] [ 2210] 0 2210 27552 31 8 0 0 agetty [314600.011749] [ 2214] 0 2214 31597 155 18 0 0 crond [314600.013590] [16982] 0 16982 29083 302 14 0 0 net_device_wait [314600.015554] [16983] 0 16983 29083 302 12 0 0 net_device_wait [314600.017492] [21402] 0 21402 38696 325 77 0 0 sshd [314600.019270] [21404] 0 21404 28953 165 11 0 0 bash [314600.021049] [27616] 89 27616 22990 257 46 0 0 pickup [314600.022847] [11782] 0 11782 27014 22 8 0 0 sleep [314600.024609] [11785] 0 11785 2168 28 10 0 0 systemd-cgroups [314600.026494] [11786] 0 11786 2168 27 10 0 0 systemd-cgroups [314600.028356] [11787] 0 11787 29083 302 12 0 0 net_device_wait [314600.030205] [11788] 0 11788 94 10 5 0 0 systemd-cgroups [314600.032047] Out of memory: Kill process 21402 (sshd) score 0 or sacrifice child [314600.033667] Killed process 21404 (bash), UID 0, total-vm:115812kB, anon-rss:656kB, file-rss:4kB, shmem-rss:0kB [314600.036981] net_device_wait invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [314600.038812] net_device_wait cpuset=/ mems_allowed=0-1 [314600.040075] CPU: 0 PID: 11787 Comm: net_device_wait Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314600.042828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314600.045551] Call Trace: [314600.046377] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314600.047631] [<ffffffff95b81668>] dump_header+0x90/0x229 [314600.048916] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314600.050254] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314600.051561] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314600.052917] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314600.054322] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314600.055630] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314600.057071] [<ffffffff956193b8>] alloc_pages_current+0x98/0x110 [314600.058459] [<ffffffff955be057>] __page_cache_alloc+0x97/0xb0 [314600.059819] [<ffffffff955c1000>] filemap_fault+0x270/0x420 [314600.061166] [<ffffffffc05fc91e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [314600.062659] [<ffffffffc05fcb1c>] xfs_filemap_fault+0x2c/0x30 [xfs] [314600.064090] [<ffffffff955ee7da>] __do_fault.isra.61+0x8a/0x100 [314600.065464] [<ffffffff955eed8c>] do_read_fault.isra.63+0x4c/0x1b0 [314600.066883] [<ffffffff955f65d0>] handle_mm_fault+0xa20/0xfb0 [314600.068225] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314600.069568] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314600.070951] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314600.072320] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314600.073647] Mem-Info: [314600.074450] active_anon:161650 inactive_anon:14403 isolated_anon:0 active_file:1 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40560 slab_unreclaimable:110713 mapped:11 shmem:172236 pagetables:658 bounce:0 free:91267 free_pcp:163 free_cma:0 [314600.082118] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314600.090683] lowmem_reserve[]: 0 2607 24634 24634 [314600.092040] Node 0 DMA32 free:101700kB min:13744kB low:17180kB high:20616kB active_anon:116kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314600.101031] lowmem_reserve[]: 0 0 22026 22026 [314600.102412] Node 0 Normal free:115952kB min:116096kB low:145120kB high:174144kB active_anon:21204kB inactive_anon:26232kB active_file:4kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:41260kB slab_reclaimable:81572kB slab_unreclaimable:198992kB kernel_stack:1616kB pagetables:816kB unstable:0kB bounce:0kB free_pcp:648kB local_pcp:28kB free_cma:0kB writeback_tmp:0kB pages_scanned:470 all_unreclaimable? yes [314600.112636] lowmem_reserve[]: 0 0 0 0 [314600.113942] Node 1 Normal free:131604kB min:132216kB low:165268kB high:198324kB active_anon:625280kB inactive_anon:31380kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:647684kB slab_reclaimable:74184kB slab_unreclaimable:217572kB kernel_stack:2160kB pagetables:1816kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? yes [314600.124118] lowmem_reserve[]: 0 0 0 0 [314600.125434] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314600.129166] Node 0 DMA32: 37*4kB (UEM) 38*8kB (EM) 16*16kB (UM) 16*32kB (UEM) 6*64kB (M) 10*128kB (UEM) 6*256kB (UEM) 6*512kB (UM) 4*1024kB (UEM) 2*2048kB (M) 21*4096kB (M) = 101700kB [314600.133533] Node 0 Normal: 1707*4kB (UEM) 1652*8kB (EM) 1062*16kB (EM) 688*32kB (UEM) 413*64kB (UEM) 192*128kB (EM) 21*256kB (UM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 115948kB [314600.137867] Node 1 Normal: 1571*4kB (UEM) 2047*8kB (EM) 1436*16kB (UEM) 1116*32kB (UEM) 763*64kB (UM) 16*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 132228kB [314600.142076] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314600.144225] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314600.146324] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314600.148463] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314600.150547] 172261 total pagecache pages [314600.151931] 0 pages in swap cache [314600.153215] Swap cache stats: add 0, delete 0, find 0/0 [314600.154805] Free swap = 0kB [314600.156030] Total swap = 0kB [314600.157254] 13055868 pages RAM [314600.158557] 0 pages HighMem/MovableOnly [314600.159959] 323772 pages reserved [314600.161525] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314600.163666] [ 618] 0 618 49643 96 30 0 0 lvmetad [314600.165717] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314600.167832] [ 801] 0 801 13883 110 29 0 -1000 auditd [314600.169851] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314600.171910] [ 861] 0 861 13212 203 30 0 0 smartd [314600.173919] [ 872] 0 872 11042 91 29 0 0 qemu-ga [314600.174911] LNet: 17020:0:(o2iblnd_cb.c:3429:kiblnd_check_conns()) Timed out tx for AA.AA.ID27@o2ibxx: 3 seconds [314600.174913] LNet: 17020:0:(o2iblnd_cb.c:3429:kiblnd_check_conns()) Skipped 207 previous similar messages [314600.180445] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314600.182483] [ 898] 38 898 7457 127 19 0 0 ntpd [314600.184446] [ 916] 0 916 6596 77 17 0 0 systemd-logind [314600.186509] [ 1949] 0 1949 22964 264 45 0 0 master [314600.188439] [ 1951] 89 1951 23007 255 48 0 0 qmgr [314600.190329] [ 2202] 0 2202 28244 258 57 0 -1000 sshd [314600.192209] [ 2208] 0 2208 27552 31 11 0 0 agetty [314600.194095] [ 2209] 0 2209 6477 50 18 0 0 atd [314600.195927] [ 2210] 0 2210 27552 31 8 0 0 agetty [314600.197788] [ 2214] 0 2214 31597 155 18 0 0 crond [314600.199612] [16982] 0 16982 29083 302 14 0 0 net_device_wait [314600.201556] [16983] 0 16983 29083 302 12 0 0 net_device_wait [314600.203504] [21402] 0 21402 38696 325 77 0 0 sshd [314600.205287] [27616] 89 27616 22990 257 46 0 0 pickup [314600.207093] [11782] 0 11782 27014 22 8 0 0 sleep [314600.208871] [11785] 0 11785 2168 28 10 0 0 systemd-cgroups [314600.210761] [11786] 0 11786 2168 27 10 0 0 systemd-cgroups [314600.212701] [11787] 0 11787 29083 302 12 0 0 net_device_wait [314600.214643] [11788] 0 11788 94 10 5 0 0 systemd-cgroups [314600.216499] Out of memory: Kill process 21402 (sshd) score 0 or sacrifice child [314600.218131] Killed process 21402 (sshd), UID 0, total-vm:154784kB, anon-rss:1300kB, file-rss:0kB, shmem-rss:0kB [314600.218133] sshd: page allocation failure: order:0, mode:0x201da [314600.218136] CPU: 8 PID: 21402 Comm: sshd Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314600.218136] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314600.218137] Call Trace: [314600.218142] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314600.218145] [<ffffffff955c4c00>] warn_alloc_failed+0x110/0x180 [314600.218147] [<ffffffff955c7920>] ? drain_pages+0xb0/0xb0 [314600.218148] [<ffffffff955c979f>] __alloc_pages_nodemask+0x9df/0xbe0 [314600.218152] [<ffffffff956193b8>] alloc_pages_current+0x98/0x110 [314600.218154] [<ffffffff955be057>] __page_cache_alloc+0x97/0xb0 [314600.218156] [<ffffffff955c1000>] filemap_fault+0x270/0x420 [314600.218184] [<ffffffffc05fc91e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [314600.218196] [<ffffffffc05fcb1c>] xfs_filemap_fault+0x2c/0x30 [xfs] [314600.218198] [<ffffffff955ee7da>] __do_fault.isra.61+0x8a/0x100 [314600.218199] [<ffffffff955eed8c>] do_read_fault.isra.63+0x4c/0x1b0 [314600.218200] [<ffffffff955f65d0>] handle_mm_fault+0xa20/0xfb0 [314600.218203] [<ffffffff954b3a15>] ? __set_current_blocked+0x55/0x70 [314600.218205] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314600.218206] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314600.218207] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314600.218208] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314600.218209] Mem-Info: [314600.218211] active_anon:161650 inactive_anon:14403 isolated_anon:0 active_file:1 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40560 slab_unreclaimable:110713 mapped:11 shmem:172236 pagetables:658 bounce:0 free:91267 free_pcp:0 free_cma:0 [314600.218214] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314600.218216] lowmem_reserve[]: 0 2607 24634 24634 [314600.218218] Node 0 DMA32 free:101700kB min:13744kB low:17180kB high:20616kB active_anon:116kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314600.218219] lowmem_reserve[]: 0 0 22026 22026 [314600.218221] Node 0 Normal free:115952kB min:116096kB low:145120kB high:174144kB active_anon:21204kB inactive_anon:26232kB active_file:4kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:41260kB slab_reclaimable:81572kB slab_unreclaimable:198992kB kernel_stack:1616kB pagetables:816kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [314600.218222] lowmem_reserve[]: 0 0 0 0 [314600.218224] Node 1 Normal free:131604kB min:132216kB low:165268kB high:198324kB active_anon:625280kB inactive_anon:31380kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:647684kB slab_reclaimable:74184kB slab_unreclaimable:217572kB kernel_stack:2160kB pagetables:1816kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314600.218225] lowmem_reserve[]: 0 0 0 0 [314600.218229] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314600.218234] Node 0 DMA32: 37*4kB (UEM) 38*8kB (EM) 16*16kB (UM) 16*32kB (UEM) 6*64kB (M) 10*128kB (UEM) 6*256kB (UEM) 6*512kB (UM) 4*1024kB (UEM) 2*2048kB (M) 21*4096kB (M) = 101700kB [314600.218238] Node 0 Normal: 1839*4kB (UEM) 1658*8kB (UEM) 1065*16kB (UEM) 689*32kB (UEM) 413*64kB (UEM) 192*128kB (EM) 21*256kB (UM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 116604kB [314600.218241] Node 1 Normal: 1567*4kB (UEM) 2047*8kB (EM) 1436*16kB (UEM) 1116*32kB (UEM) 763*64kB (UM) 16*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 132212kB [314600.218242] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314600.218243] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314600.218243] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314600.218244] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314600.218244] 172261 total pagecache pages [314600.218245] 0 pages in swap cache [314600.218245] Swap cache stats: add 0, delete 0, find 0/0 [314600.218246] Free swap = 0kB [314600.218246] Total swap = 0kB [314600.218246] 13055868 pages RAM [314600.218247] 0 pages HighMem/MovableOnly [314600.218247] 323772 pages reserved [314600.337880] Out of memory: Kill process 16982 (net_device_wait) score 0 or sacrifice child [314600.340331] Killed process 11787 (net_device_wait), UID 0, total-vm:116332kB, anon-rss:1208kB, file-rss:0kB, shmem-rss:0kB [314600.344011] Out of memory: Kill process 16982 (net_device_wait) score 0 or sacrifice child [314600.345886] Killed process 16982 (net_device_wait), UID 0, total-vm:116332kB, anon-rss:1208kB, file-rss:0kB, shmem-rss:0kB [314600.348993] Out of memory: Kill process 16983 (net_device_wait) score 0 or sacrifice child [314600.350835] Killed process 11782 (sleep), UID 0, total-vm:108056kB, anon-rss:88kB, file-rss:0kB, shmem-rss:0kB [314600.353997] Out of memory: Kill process 16983 (net_device_wait) score 0 or sacrifice child [314600.355803] Killed process 16983 (net_device_wait), UID 0, total-vm:116332kB, anon-rss:1208kB, file-rss:0kB, shmem-rss:0kB [314600.358964] Out of memory: Kill process 1951 (qmgr) score 0 or sacrifice child [314600.360788] Killed process 1951 (qmgr), UID 89, total-vm:92028kB, anon-rss:1020kB, file-rss:0kB, shmem-rss:0kB [314600.364356] Out of memory: Kill process 27616 (pickup) score 0 or sacrifice child [314600.366270] Killed process 27616 (pickup), UID 89, total-vm:91960kB, anon-rss:1028kB, file-rss:0kB, shmem-rss:0kB [314600.371705] Out of memory: Kill process 1949 (master) score 0 or sacrifice child [314600.373643] Killed process 1949 (master), UID 0, total-vm:91856kB, anon-rss:1052kB, file-rss:0kB, shmem-rss:0kB [314600.833952] Out of memory: Kill process 861 (smartd) score 0 or sacrifice child [314600.837720] Killed process 861 (smartd), UID 0, total-vm:52848kB, anon-rss:812kB, file-rss:0kB, shmem-rss:0kB [314600.841167] Out of memory: Kill process 2214 (crond) score 0 or sacrifice child [314600.842898] Killed process 2214 (crond), UID 0, total-vm:126388kB, anon-rss:620kB, file-rss:0kB, shmem-rss:0kB [314600.847255] Out of memory: Kill process 898 (ntpd) score 0 or sacrifice child [314600.849334] Killed process 898 (ntpd), UID 38, total-vm:29828kB, anon-rss:504kB, file-rss:4kB, shmem-rss:0kB [314600.858035] Out of memory: Kill process 618 (lvmetad) score 0 or sacrifice child [314600.859778] Killed process 618 (lvmetad), UID 0, total-vm:198572kB, anon-rss:384kB, file-rss:0kB, shmem-rss:0kB [314603.174020] LNet: 17020:0:(o2iblnd_cb.c:3429:kiblnd_check_conns()) Timed out tx for BB.BB.ID17@o2ibyy: 6 seconds [314603.178089] LNet: 17020:0:(o2iblnd_cb.c:3429:kiblnd_check_conns()) Skipped 233 previous similar messages [314603.892118] oom_kill_process: 11 callbacks suppressed [314603.893857] systemd-cgroups invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 [314603.895602] systemd-cgroups cpuset=/ mems_allowed=0-1 [314603.896970] CPU: 7 PID: 11799 Comm: systemd-cgroups Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314603.899886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314603.902829] Call Trace: [314603.903897] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314603.905260] [<ffffffff95b81668>] dump_header+0x90/0x229 [314603.906656] [<ffffffff95b8ea15>] ? _raw_spin_unlock_irqrestore+0x15/0x20 [314603.908220] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314603.909657] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314603.911144] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314603.912561] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314603.914089] [<ffffffff9561cc49>] alloc_pages_vma+0xa9/0x200 [314603.915503] [<ffffffff955f6867>] handle_mm_fault+0xcb7/0xfb0 [314603.916929] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314603.918346] [<ffffffff955fa3a8>] ? __vma_link_rb+0xb8/0xe0 [314603.919737] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314603.921196] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314603.922643] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314603.924060] [<ffffffff95795609>] ? copy_user_enhanced_fast_string+0x9/0x20 [314603.925629] [<ffffffff95b83aab>] ? create_elf_tables+0x9d/0x56d [314603.927081] [<ffffffff955fdba4>] ? install_special_mapping+0xe4/0x100 [314603.928581] [<ffffffff956b533a>] load_elf_binary+0xa7a/0xdb0 [314603.929972] [<ffffffff9565573a>] search_binary_handler+0x9a/0x1c0 [314603.931407] [<ffffffff95656c26>] do_execve_common.isra.23+0x616/0x880 [314603.932865] [<ffffffff95656ea8>] do_execve+0x18/0x20 [314603.934146] [<ffffffff954ba8ff>] ____call_usermodehelper+0xff/0x140 [314603.935568] [<ffffffff954ba940>] ? ____call_usermodehelper+0x140/0x140 [314603.937007] [<ffffffff954ba95e>] call_helper+0x1e/0x20 [314603.938263] [<ffffffff95b99ddd>] ret_from_fork_nospec_begin+0x7/0x21 [314603.939666] [<ffffffff954ba940>] ? ____call_usermodehelper+0x140/0x140 [314603.941088] Mem-Info: [314603.941960] active_anon:159298 inactive_anon:14403 isolated_anon:0 active_file:1 inactive_file:12 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40561 slab_unreclaimable:110869 mapped:20 shmem:172236 pagetables:308 bounce:0 free:91328 free_pcp:0 free_cma:0 [314603.949153] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314603.956238] lowmem_reserve[]: 0 2607 24634 24634 [314603.957445] Node 0 DMA32 free:101816kB min:13744kB low:17180kB high:20616kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314603.964622] lowmem_reserve[]: 0 0 22026 22026 [314603.965767] Node 0 Normal free:115920kB min:116096kB low:145120kB high:174144kB active_anon:17644kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:41260kB slab_reclaimable:81572kB slab_unreclaimable:199044kB kernel_stack:1696kB pagetables:332kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314603.973611] lowmem_reserve[]: 0 0 0 0 [314603.974764] Node 1 Normal free:131764kB min:132216kB low:165268kB high:198324kB active_anon:619548kB inactive_anon:31380kB active_file:4kB inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:80kB shmem:647684kB slab_reclaimable:74188kB slab_unreclaimable:218144kB kernel_stack:2144kB pagetables:900kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [314603.983124] lowmem_reserve[]: 0 0 0 0 [314603.984359] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314603.987697] Node 0 DMA32: 22*4kB (UE) 20*8kB (EM) 2*16kB (UM) 3*32kB (UE) 1*64kB (M) 2*128kB (UE) 3*256kB (UEM) 2*512kB (UM) 3*1024kB (UEM) 1*2048kB (M) 23*4096kB (M) = 101816kB [314603.991519] Node 0 Normal: 1639*4kB (EM) 1529*8kB (UEM) 1036*16kB (UEM) 698*32kB (UEM) 424*64kB (UEM) 208*128kB (UEM) 17*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115812kB [314603.995334] Node 1 Normal: 1681*4kB (UEM) 1991*8kB (EM) 1415*16kB (UEM) 1096*32kB (UEM) 772*64kB (UM) 15*128kB (M) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 131948kB [314603.999149] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314604.001030] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314604.002854] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314604.004703] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314604.006511] 172270 total pagecache pages [314604.007797] 0 pages in swap cache [314604.009014] Swap cache stats: add 0, delete 0, find 0/0 [314604.010456] Free swap = 0kB [314604.011624] Total swap = 0kB [314604.012780] 13055868 pages RAM [314604.013953] 0 pages HighMem/MovableOnly [314604.015224] 323772 pages reserved [314604.016425] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314604.018187] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314604.020038] [ 801] 0 801 13883 110 29 0 -1000 auditd [314604.021791] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314604.023587] [ 872] 0 872 11042 91 29 0 0 qemu-ga [314604.025363] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314604.027159] [ 916] 0 916 6596 77 17 0 0 systemd-logind [314604.028961] [ 2202] 0 2202 28244 257 57 0 -1000 sshd [314604.030656] [ 2208] 0 2208 27552 31 11 0 0 agetty [314604.032358] [ 2209] 0 2209 6477 50 18 0 0 atd [314604.034036] [ 2210] 0 2210 27552 31 8 0 0 agetty [314604.035724] [11788] 0 11788 1168 13 7 0 0 systemd-cgroups [314604.037462] [11792] 0 11792 628 9 6 0 0 systemd-cgroups [314604.039216] [11793] 0 11793 628 10 6 0 0 systemd-cgroups [314604.040995] [11794] 0 11794 628 8 6 0 0 systemd-cgroups [314604.042691] [11795] 0 11795 628 9 6 0 0 systemd-cgroups [314604.044373] [11796] 0 11796 82 1 5 0 0 systemd-cgroups [314604.046048] [11798] 0 11798 82 1 4 0 0 systemd-cgroups [314604.047692] [11799] 0 11799 82 1 3 0 0 systemd-cgroups [314604.049326] [11800] 0 11800 79 1 2 0 0 systemd-cgroups [314604.050962] Out of memory: Kill process 872 (qemu-ga) score 0 or sacrifice child [314604.052406] Killed process 872 (qemu-ga), UID 0, total-vm:44168kB, anon-rss:364kB, file-rss:0kB, shmem-rss:0kB [314604.052449] qemu-ga: page allocation failure: order:0, mode:0x201da [314604.052452] CPU: 1 PID: 872 Comm: qemu-ga Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314604.052452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314604.052453] Call Trace: [314604.052459] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314604.052462] [<ffffffff955c4c00>] warn_alloc_failed+0x110/0x180 [314604.052463] [<ffffffff955c7920>] ? drain_pages+0xb0/0xb0 [314604.052465] [<ffffffff955c979f>] __alloc_pages_nodemask+0x9df/0xbe0 [314604.052469] [<ffffffff956193b8>] alloc_pages_current+0x98/0x110 [314604.052471] [<ffffffff955be057>] __page_cache_alloc+0x97/0xb0 [314604.052473] [<ffffffff955c1000>] filemap_fault+0x270/0x420 [314604.052512] [<ffffffffc05fc91e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [314604.052523] [<ffffffffc05fcb1c>] xfs_filemap_fault+0x2c/0x30 [xfs] [314604.052525] [<ffffffff955ee7da>] __do_fault.isra.61+0x8a/0x100 [314604.052526] [<ffffffff955eed8c>] do_read_fault.isra.63+0x4c/0x1b0 [314604.052528] [<ffffffff955f65d0>] handle_mm_fault+0xa20/0xfb0 [314604.052540] [<ffffffffc021ae9a>] ? virtqueue_kick+0x1a/0x50 [virtio_ring] [314604.052542] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314604.052543] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314604.052544] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314604.052546] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314604.052546] Mem-Info: [314604.052549] active_anon:159298 inactive_anon:14403 isolated_anon:0 active_file:1 inactive_file:12 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40561 slab_unreclaimable:110869 mapped:20 shmem:172236 pagetables:308 bounce:0 free:91328 free_pcp:0 free_cma:0 [314604.052552] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.052554] lowmem_reserve[]: 0 2607 24634 24634 [314604.052556] Node 0 DMA32 free:101816kB min:13744kB low:17180kB high:20616kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.052557] lowmem_reserve[]: 0 0 22026 22026 [314604.052559] Node 0 Normal free:115920kB min:116096kB low:145120kB high:174144kB active_anon:17644kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:41260kB slab_reclaimable:81572kB slab_unreclaimable:199044kB kernel_stack:1696kB pagetables:332kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.052560] lowmem_reserve[]: 0 0 0 0 [314604.052562] Node 1 Normal free:131764kB min:132216kB low:165268kB high:198324kB active_anon:619548kB inactive_anon:31380kB active_file:4kB inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:80kB shmem:647684kB slab_reclaimable:74188kB slab_unreclaimable:218144kB kernel_stack:2144kB pagetables:900kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [314604.052563] lowmem_reserve[]: 0 0 0 0 [314604.052567] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314604.052572] Node 0 DMA32: 22*4kB (UE) 20*8kB (EM) 2*16kB (UM) 3*32kB (UE) 1*64kB (M) 2*128kB (UE) 3*256kB (UEM) 2*512kB (UM) 3*1024kB (UEM) 1*2048kB (M) 23*4096kB (M) = 101816kB [314604.052575] Node 0 Normal: 1639*4kB (EM) 1529*8kB (UEM) 1036*16kB (UEM) 698*32kB (UEM) 424*64kB (UEM) 208*128kB (UEM) 17*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115812kB [314604.052579] Node 1 Normal: 1681*4kB (UEM) 1991*8kB (EM) 1415*16kB (UEM) 1096*32kB (UEM) 772*64kB (UM) 15*128kB (M) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 131948kB [314604.052580] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314604.052580] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314604.052581] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314604.052581] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314604.052582] 172270 total pagecache pages [314604.052582] 0 pages in swap cache [314604.052583] Swap cache stats: add 0, delete 0, find 0/0 [314604.052583] Free swap = 0kB [314604.052583] Total swap = 0kB [314604.052584] 13055868 pages RAM [314604.052584] 0 pages HighMem/MovableOnly [314604.052585] 323772 pages reserved [314604.182841] systemd-cgroups invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 [314604.185274] systemd-cgroups cpuset=/ mems_allowed=0-1 [314604.188224] CPU: 0 PID: 11801 Comm: systemd-cgroups Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314604.191691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314604.195159] Call Trace: [314604.196341] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314604.197919] [<ffffffff95b81668>] dump_header+0x90/0x229 [314604.199511] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314604.201165] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314604.202769] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314604.204468] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314604.206183] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314604.207744] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314604.209432] [<ffffffff9561cc49>] alloc_pages_vma+0xa9/0x200 [314604.211002] [<ffffffff955f6867>] handle_mm_fault+0xcb7/0xfb0 [314604.212599] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314604.214156] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314604.215755] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314604.217292] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314604.218786] Mem-Info: [314604.219736] active_anon:159298 inactive_anon:14403 isolated_anon:0 active_file:1 inactive_file:4 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40561 slab_unreclaimable:110949 mapped:20 shmem:172236 pagetables:308 bounce:0 free:91430 free_pcp:0 free_cma:0 [314604.227990] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.236507] lowmem_reserve[]: 0 2607 24634 24634 [314604.237845] Node 0 DMA32 free:101816kB min:13744kB low:17180kB high:20616kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.246795] lowmem_reserve[]: 0 0 22026 22026 [314604.248208] Node 0 Normal free:115920kB min:116096kB low:145120kB high:174144kB active_anon:17644kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:41260kB slab_reclaimable:81572kB slab_unreclaimable:199044kB kernel_stack:1696kB pagetables:332kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.258561] lowmem_reserve[]: 0 0 0 0 [314604.259890] Node 1 Normal free:132172kB min:132216kB low:165268kB high:198324kB active_anon:619548kB inactive_anon:31380kB active_file:4kB inactive_file:16kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:80kB shmem:647684kB slab_reclaimable:74188kB slab_unreclaimable:218464kB kernel_stack:2144kB pagetables:900kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [314604.270473] lowmem_reserve[]: 0 0 0 0 [314604.271818] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314604.275569] Node 0 DMA32: 22*4kB (UE) 20*8kB (EM) 2*16kB (UM) 3*32kB (UE) 1*64kB (M) 2*128kB (UE) 3*256kB (UEM) 2*512kB (UM) 3*1024kB (UEM) 1*2048kB (M) 23*4096kB (M) = 101816kB [314604.279834] Node 0 Normal: 1639*4kB (UEM) 1529*8kB (UEM) 1034*16kB (UEM) 696*32kB (UEM) 423*64kB (UEM) 210*128kB (UEM) 17*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115908kB [314604.284109] Node 1 Normal: 1650*4kB (UEM) 2027*8kB (UEM) 1421*16kB (UEM) 1099*32kB (UEM) 775*64kB (UM) 15*128kB (M) 0*256kB 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 132752kB [314604.288405] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314604.290542] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314604.292634] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314604.294761] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314604.296835] 172270 total pagecache pages [314604.298238] 0 pages in swap cache [314604.299525] Swap cache stats: add 0, delete 0, find 0/0 [314604.301135] Free swap = 0kB [314604.302352] Total swap = 0kB [314604.303572] 13055868 pages RAM [314604.304814] 0 pages HighMem/MovableOnly [314604.306181] 323772 pages reserved [314604.307455] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314604.309463] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314604.311583] [ 801] 0 801 13883 110 29 0 -1000 auditd [314604.313598] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314604.315656] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314604.317728] [ 916] 0 916 6596 77 17 0 0 systemd-logind [314604.319841] [ 2202] 0 2202 28244 257 57 0 -1000 sshd [314604.321809] [ 2208] 0 2208 27552 31 11 0 0 agetty [314604.323788] [ 2209] 0 2209 6477 50 18 0 0 atd [314604.325704] [ 2210] 0 2210 27552 31 8 0 0 agetty [314604.327660] [11788] 0 11788 1168 14 7 0 0 systemd-cgroups [314604.329629] [11792] 0 11792 1168 12 7 0 0 systemd-cgroups [314604.331711] [11793] 0 11793 1168 10 7 0 0 systemd-cgroups [314604.333723] [11794] 0 11794 1168 11 7 0 0 systemd-cgroups [314604.335701] [11795] 0 11795 628 12 6 0 0 systemd-cgroups [314604.337664] [11796] 0 11796 628 8 6 0 0 systemd-cgroups [314604.339609] [11798] 0 11798 1168 6 7 0 0 systemd-cgroups [314604.341549] [11799] 0 11799 1168 1 7 0 0 systemd-cgroups [314604.343463] [11800] 0 11800 1168 1 7 0 0 systemd-cgroups [314604.345373] [11801] 0 11801 82 1 4 0 0 systemd-cgroups [314604.347261] Out of memory: Kill process 916 (systemd-logind) score 0 or sacrifice child [314604.349040] Killed process 916 (systemd-logind), UID 0, total-vm:26384kB, anon-rss:308kB, file-rss:0kB, shmem-rss:0kB [314604.349047] systemd-logind: page allocation failure: order:0, mode:0x201da [314604.349049] CPU: 4 PID: 916 Comm: systemd-logind Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314604.349049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314604.349050] Call Trace: [314604.349053] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314604.349055] [<ffffffff955c4c00>] warn_alloc_failed+0x110/0x180 [314604.349056] [<ffffffff955c7920>] ? drain_pages+0xb0/0xb0 [314604.349057] [<ffffffff955c979f>] __alloc_pages_nodemask+0x9df/0xbe0 [314604.349059] [<ffffffff956193b8>] alloc_pages_current+0x98/0x110 [314604.349061] [<ffffffff955be057>] __page_cache_alloc+0x97/0xb0 [314604.349063] [<ffffffff955c1000>] filemap_fault+0x270/0x420 [314604.349098] [<ffffffffc05fc91e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [314604.349109] [<ffffffffc05fcb1c>] xfs_filemap_fault+0x2c/0x30 [xfs] [314604.349110] [<ffffffff955ee7da>] __do_fault.isra.61+0x8a/0x100 [314604.349112] [<ffffffff955eed8c>] do_read_fault.isra.63+0x4c/0x1b0 [314604.349113] [<ffffffff955f65d0>] handle_mm_fault+0xa20/0xfb0 [314604.349115] [<ffffffff955d727d>] ? shmem_destroy_inode+0x2d/0x40 [314604.349117] [<ffffffff9566c61b>] ? destroy_inode+0x3b/0x60 [314604.349118] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314604.349120] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314604.349121] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314604.349121] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314604.349122] Mem-Info: [314604.349124] active_anon:159298 inactive_anon:14403 isolated_anon:0 active_file:1 inactive_file:4 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40561 slab_unreclaimable:110949 mapped:20 shmem:172236 pagetables:308 bounce:0 free:91430 free_pcp:0 free_cma:0 [314604.349127] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.349128] lowmem_reserve[]: 0 2607 24634 24634 [314604.349130] Node 0 DMA32 free:101816kB min:13744kB low:17180kB high:20616kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.349131] lowmem_reserve[]: 0 0 22026 22026 [314604.349133] Node 0 Normal free:115920kB min:116096kB low:145120kB high:174144kB active_anon:17644kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:41260kB slab_reclaimable:81572kB slab_unreclaimable:199044kB kernel_stack:1696kB pagetables:332kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.349134] lowmem_reserve[]: 0 0 0 0 [314604.349136] Node 1 Normal free:132172kB min:132216kB low:165268kB high:198324kB active_anon:619548kB inactive_anon:31380kB active_file:4kB inactive_file:16kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:80kB shmem:647684kB slab_reclaimable:74188kB slab_unreclaimable:218464kB kernel_stack:2144kB pagetables:900kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [314604.349137] lowmem_reserve[]: 0 0 0 0 [314604.349141] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314604.349145] Node 0 DMA32: 22*4kB (UE) 20*8kB (EM) 2*16kB (UM) 3*32kB (UE) 1*64kB (M) 2*128kB (UE) 3*256kB (UEM) 2*512kB (UM) 3*1024kB (UEM) 1*2048kB (M) 23*4096kB (M) = 101816kB [314604.349149] Node 0 Normal: 1639*4kB (UEM) 1529*8kB (UEM) 1034*16kB (UEM) 696*32kB (UEM) 423*64kB (UEM) 210*128kB (UEM) 17*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115908kB [314604.349153] Node 1 Normal: 1650*4kB (UEM) 2027*8kB (UEM) 1421*16kB (UEM) 1099*32kB (UEM) 775*64kB (UM) 15*128kB (M) 0*256kB 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 132752kB [314604.349153] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314604.349154] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314604.349154] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314604.349155] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314604.349155] 172270 total pagecache pages [314604.349155] 0 pages in swap cache [314604.349156] Swap cache stats: add 0, delete 0, find 0/0 [314604.349156] Free swap = 0kB [314604.349156] Total swap = 0kB [314604.349157] 13055868 pages RAM [314604.349157] 0 pages HighMem/MovableOnly [314604.349158] 323772 pages reserved [314604.562161] kworker/2:1 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [314604.564198] kworker/2:1 cpuset=/ mems_allowed=0-1 [314604.565498] CPU: 2 PID: 15808 Comm: kworker/2:1 Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314604.568360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314604.571274] Workqueue: ib_cm cm_work_handler [ib_cm] [314604.572640] Call Trace: [314604.573687] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314604.575036] [<ffffffff95b81668>] dump_header+0x90/0x229 [314604.576385] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314604.577771] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314604.579129] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314604.580502] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314604.581896] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314604.583217] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314604.584633] [<ffffffff95606d5f>] __vmalloc_node_range+0x12f/0x280 [314604.585995] [<ffffffffc0925407>] ? kiblnd_create_conn+0xa87/0x1a00 [ko2iblnd] [314604.587463] [<ffffffff95606f4e>] vzalloc_node+0x4e/0x50 [314604.588714] [<ffffffffc0925407>] ? kiblnd_create_conn+0xa87/0x1a00 [ko2iblnd] [314604.590174] [<ffffffffc0925407>] kiblnd_create_conn+0xa87/0x1a00 [ko2iblnd] [314604.591613] [<ffffffff95628a0d>] ? kmem_cache_alloc_node_trace+0x11d/0x210 [314604.593025] [<ffffffffc0932faf>] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [314604.594475] [<ffffffffc0934445>] kiblnd_cm_callback+0x755/0x23a0 [ko2iblnd] [314604.595890] [<ffffffffc06fff18>] ? rdma_restrack_add+0x98/0x130 [ib_core] [314604.597269] [<ffffffffc07ba62f>] cma_ib_req_handler+0x61f/0x1280 [rdma_cm] [314604.598635] [<ffffffffc04e23eb>] cm_process_work+0x2b/0x130 [ib_cm] [314604.599938] [<ffffffffc04e2538>] cm_queue_work_unlock+0x48/0x90 [ib_cm] [314604.601262] [<ffffffffc04e3787>] cm_req_handler+0xb47/0xfe0 [ib_cm] [314604.602539] [<ffffffffc04e23f6>] ? cm_process_work+0x36/0x130 [ib_cm] [314604.603816] [<ffffffffc04e47a5>] cm_work_handler+0x155/0x135d [ib_cm] [314604.605086] [<ffffffff954bb42e>] ? move_linked_works+0x5e/0x90 [314604.606264] [<ffffffff954bbb5f>] ? pwq_activate_delayed_work+0x3f/0xb0 [314604.607529] [<ffffffff954bdfbf>] process_one_work+0x17f/0x440 [314604.608682] [<ffffffff954bf0d6>] worker_thread+0x126/0x3c0 [314604.609776] [<ffffffff954befb0>] ? manage_workers.isra.26+0x2a0/0x2a0 [314604.610977] [<ffffffff954c5f91>] kthread+0xd1/0xe0 [314604.611988] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314604.613148] [<ffffffff95b99ddd>] ret_from_fork_nospec_begin+0x7/0x21 [314604.614356] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314604.615516] Mem-Info: [314604.616243] active_anon:159298 inactive_anon:14403 isolated_anon:0 active_file:1 inactive_file:4 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40561 slab_unreclaimable:110949 mapped:20 shmem:172236 pagetables:308 bounce:0 free:91398 free_pcp:1 free_cma:0 [314604.622674] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.629351] lowmem_reserve[]: 0 2607 24634 24634 [314604.630460] Node 0 DMA32 free:101816kB min:13744kB low:17180kB high:20616kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.637383] lowmem_reserve[]: 0 0 22026 22026 [314604.638514] Node 0 Normal free:115920kB min:116096kB low:145120kB high:174144kB active_anon:17644kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:41260kB slab_reclaimable:81572kB slab_unreclaimable:199044kB kernel_stack:1696kB pagetables:332kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.646516] lowmem_reserve[]: 0 0 0 0 [314604.647632] Node 1 Normal free:132044kB min:132216kB low:165268kB high:198324kB active_anon:619548kB inactive_anon:31380kB active_file:4kB inactive_file:16kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:80kB shmem:647684kB slab_reclaimable:74188kB slab_unreclaimable:218464kB kernel_stack:2144kB pagetables:900kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:155 all_unreclaimable? yes [314604.656005] lowmem_reserve[]: 0 0 0 0 [314604.657188] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314604.660378] Node 0 DMA32: 22*4kB (UE) 20*8kB (EM) 2*16kB (UM) 3*32kB (UE) 1*64kB (M) 2*128kB (UE) 3*256kB (UEM) 2*512kB (UM) 3*1024kB (UEM) 1*2048kB (M) 23*4096kB (M) = 101816kB [314604.663998] Node 0 Normal: 1645*4kB (UEM) 1528*8kB (UEM) 1033*16kB (UEM) 694*32kB (UEM) 424*64kB (UEM) 211*128kB (UEM) 17*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 116036kB [314604.667600] Node 1 Normal: 1640*4kB (UEM) 2002*8kB (UEM) 1417*16kB (UM) 1099*32kB (UEM) 774*64kB (UM) 16*128kB (M) 0*256kB 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 132512kB [314604.671216] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314604.672994] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314604.674731] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314604.676506] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314604.678252] 172270 total pagecache pages [314604.679476] 0 pages in swap cache [314604.680628] Swap cache stats: add 0, delete 0, find 0/0 [314604.681989] Free swap = 0kB [314604.683084] Total swap = 0kB [314604.684179] 13055868 pages RAM [314604.685289] 0 pages HighMem/MovableOnly [314604.686484] 323772 pages reserved [314604.687614] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314604.689282] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314604.691040] [ 801] 0 801 13883 110 29 0 -1000 auditd [314604.692713] [ 854] 0 854 2881 68 11 0 0 rasdaemon [314604.694433] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314604.696163] [ 2202] 0 2202 28244 257 57 0 -1000 sshd [314604.697812] [ 2208] 0 2208 27552 31 11 0 0 agetty [314604.699462] [ 2209] 0 2209 6477 50 18 0 0 atd [314604.701072] [ 2210] 0 2210 27552 31 8 0 0 agetty [314604.702692] [11788] 0 11788 1168 14 7 0 0 systemd-cgroups [314604.704391] [11792] 0 11792 1168 12 7 0 0 systemd-cgroups [314604.706076] [11793] 0 11793 1168 13 7 0 0 systemd-cgroups [314604.707735] [11794] 0 11794 1168 11 7 0 0 systemd-cgroups [314604.709387] [11795] 0 11795 628 12 6 0 0 systemd-cgroups [314604.711021] [11796] 0 11796 628 8 6 0 0 systemd-cgroups [314604.712627] [11798] 0 11798 1168 12 7 0 0 systemd-cgroups [314604.714230] [11799] 0 11799 1168 1 7 0 0 systemd-cgroups [314604.715810] [11800] 0 11800 1168 1 7 0 0 systemd-cgroups [314604.717366] [11801] 0 11801 82 1 4 0 0 systemd-cgroups [314604.718911] Out of memory: Kill process 854 (rasdaemon) score 0 or sacrifice child [314604.720326] Killed process 854 (rasdaemon), UID 0, total-vm:11524kB, anon-rss:272kB, file-rss:0kB, shmem-rss:0kB [314604.723669] kworker/2:1 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [314604.725122] kworker/2:1 cpuset=/ mems_allowed=0-1 [314604.726191] CPU: 2 PID: 15808 Comm: kworker/2:1 Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314604.728566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314604.730959] Workqueue: ib_cm cm_work_handler [ib_cm] [314604.732084] Call Trace: [314604.732865] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314604.733953] [<ffffffff95b81668>] dump_header+0x90/0x229 [314604.735055] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314604.736176] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314604.737274] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314604.738384] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314604.739521] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314604.740578] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314604.741729] [<ffffffff95606d5f>] __vmalloc_node_range+0x12f/0x280 [314604.742863] [<ffffffffc0925407>] ? kiblnd_create_conn+0xa87/0x1a00 [ko2iblnd] [314604.744124] [<ffffffff95606f4e>] vzalloc_node+0x4e/0x50 [314604.745163] [<ffffffffc0925407>] ? kiblnd_create_conn+0xa87/0x1a00 [ko2iblnd] [314604.746427] [<ffffffffc0925407>] kiblnd_create_conn+0xa87/0x1a00 [ko2iblnd] [314604.747670] [<ffffffff95628a0d>] ? kmem_cache_alloc_node_trace+0x11d/0x210 [314604.748903] [<ffffffffc0932faf>] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [314604.750202] [<ffffffffc0934445>] kiblnd_cm_callback+0x755/0x23a0 [ko2iblnd] [314604.751478] [<ffffffffc06fff18>] ? rdma_restrack_add+0x98/0x130 [ib_core] [314604.752716] [<ffffffffc07ba62f>] cma_ib_req_handler+0x61f/0x1280 [rdma_cm] [314604.753968] [<ffffffffc04e23eb>] cm_process_work+0x2b/0x130 [ib_cm] [314604.755161] [<ffffffffc04e2538>] cm_queue_work_unlock+0x48/0x90 [ib_cm] [314604.756381] [<ffffffffc04e3787>] cm_req_handler+0xb47/0xfe0 [ib_cm] [314604.757553] [<ffffffffc04e23f6>] ? cm_process_work+0x36/0x130 [ib_cm] [314604.758752] [<ffffffffc04e47a5>] cm_work_handler+0x155/0x135d [ib_cm] [314604.759946] [<ffffffff954bb42e>] ? move_linked_works+0x5e/0x90 [314604.761073] [<ffffffff954bbb5f>] ? pwq_activate_delayed_work+0x3f/0xb0 [314604.762280] [<ffffffff954bdfbf>] process_one_work+0x17f/0x440 [314604.763406] [<ffffffff954bf0d6>] worker_thread+0x126/0x3c0 [314604.764505] [<ffffffff954befb0>] ? manage_workers.isra.26+0x2a0/0x2a0 [314604.765701] [<ffffffff954c5f91>] kthread+0xd1/0xe0 [314604.766716] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314604.767874] [<ffffffff95b99ddd>] ret_from_fork_nospec_begin+0x7/0x21 [314604.769090] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314604.770261] Mem-Info: [314604.770994] active_anon:159225 inactive_anon:14403 isolated_anon:0 active_file:1 inactive_file:4 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40561 slab_unreclaimable:110949 mapped:20 shmem:172236 pagetables:308 bounce:0 free:91398 free_pcp:106 free_cma:0 [314604.777572] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.784244] lowmem_reserve[]: 0 2607 24634 24634 [314604.785362] Node 0 DMA32 free:101816kB min:13744kB low:17180kB high:20616kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.792290] lowmem_reserve[]: 0 0 22026 22026 [314604.793423] Node 0 Normal free:115920kB min:116096kB low:145120kB high:174144kB active_anon:17352kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:41260kB slab_reclaimable:81572kB slab_unreclaimable:199044kB kernel_stack:1696kB pagetables:332kB unstable:0kB bounce:0kB free_pcp:420kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.801445] lowmem_reserve[]: 0 0 0 0 [314604.802559] Node 1 Normal free:132044kB min:132216kB low:165268kB high:198324kB active_anon:619548kB inactive_anon:31380kB active_file:4kB inactive_file:16kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:80kB shmem:647684kB slab_reclaimable:74188kB slab_unreclaimable:218464kB kernel_stack:2144kB pagetables:900kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:891 all_unreclaimable? yes [314604.810933] lowmem_reserve[]: 0 0 0 0 [314604.812129] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314604.815311] Node 0 DMA32: 22*4kB (UE) 20*8kB (EM) 2*16kB (UM) 3*32kB (UE) 1*64kB (M) 2*128kB (UE) 3*256kB (UEM) 2*512kB (UM) 3*1024kB (UEM) 1*2048kB (M) 23*4096kB (M) = 101816kB [314604.818922] Node 0 Normal: 1620*4kB (EM) 1526*8kB (UEM) 1035*16kB (UEM) 694*32kB (UEM) 426*64kB (UEM) 211*128kB (UEM) 17*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 116080kB [314604.822540] Node 1 Normal: 1640*4kB (UEM) 2002*8kB (UEM) 1417*16kB (UM) 1099*32kB (UEM) 774*64kB (UM) 16*128kB (M) 0*256kB 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 132512kB [314604.826150] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314604.827938] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314604.829675] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314604.831438] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314604.833163] 172270 total pagecache pages [314604.834379] 0 pages in swap cache [314604.835526] Swap cache stats: add 0, delete 0, find 0/0 [314604.836895] Free swap = 0kB [314604.837987] Total swap = 0kB [314604.840070] 13055868 pages RAM [314604.841178] 0 pages HighMem/MovableOnly [314604.842377] 323772 pages reserved [314604.843509] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314604.845183] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314604.846943] [ 801] 0 801 13883 110 29 0 -1000 auditd [314604.848631] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314604.850366] [ 2202] 0 2202 28244 257 57 0 -1000 sshd [314604.852125] [ 2208] 0 2208 27552 31 11 0 0 agetty [314604.853802] [ 2209] 0 2209 6477 50 18 0 0 atd [314604.855431] [ 2210] 0 2210 27552 31 8 0 0 agetty [314604.857079] [11788] 0 11788 1168 14 7 0 0 systemd-cgroups [314604.858946] [11792] 0 11792 1168 12 7 0 0 systemd-cgroups [314604.860646] [11793] 0 11793 1168 13 7 0 0 systemd-cgroups [314604.862327] [11794] 0 11794 1168 11 7 0 0 systemd-cgroups [314604.865400] [11795] 0 11795 628 12 6 0 0 systemd-cgroups [314604.867053] [11796] 0 11796 628 8 6 0 0 systemd-cgroups [314604.868685] [11798] 0 11798 1168 12 7 0 0 systemd-cgroups [314604.870290] [11799] 0 11799 1168 13 7 0 0 systemd-cgroups [314604.871886] [11800] 0 11800 1168 1 7 0 0 systemd-cgroups [314604.873460] [11801] 0 11801 82 1 5 0 0 systemd-cgroups [314604.875016] Out of memory: Kill process 2209 (atd) score 0 or sacrifice child [314604.876390] Killed process 2209 (atd), UID 0, total-vm:25908kB, anon-rss:200kB, file-rss:0kB, shmem-rss:0kB [314604.886145] systemd-cgroups invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 [314604.888583] systemd-cgroups cpuset=/ mems_allowed=0-1 [314604.890006] CPU: 5 PID: 11788 Comm: systemd-cgroups Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314604.893044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314604.895994] Call Trace: [314604.896931] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314604.898277] [<ffffffff95b81668>] dump_header+0x90/0x229 [314604.899652] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314604.901134] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314604.902583] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314604.904029] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314604.905456] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314604.906777] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314604.908289] [<ffffffff9561cc49>] alloc_pages_vma+0xa9/0x200 [314604.909617] [<ffffffff955f6867>] handle_mm_fault+0xcb7/0xfb0 [314604.910942] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314604.912279] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314604.913645] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314604.914995] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314604.916305] Mem-Info: [314604.917084] active_anon:159036 inactive_anon:14403 isolated_anon:0 active_file:6 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40566 slab_unreclaimable:110954 mapped:14 shmem:172236 pagetables:250 bounce:0 free:91356 free_pcp:0 free_cma:0 [314604.924682] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.933004] lowmem_reserve[]: 0 2607 24634 24634 [314604.934314] Node 0 DMA32 free:101816kB min:13744kB low:17180kB high:20616kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314604.943535] lowmem_reserve[]: 0 0 22026 22026 [314604.944933] Node 0 Normal free:116024kB min:116096kB low:145120kB high:174144kB active_anon:17128kB inactive_anon:26232kB active_file:8kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:41260kB slab_reclaimable:81572kB slab_unreclaimable:198948kB kernel_stack:1696kB pagetables:288kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [314604.955216] lowmem_reserve[]: 0 0 0 0 [314604.956537] Node 1 Normal free:131976kB min:132216kB low:165268kB high:198324kB active_anon:619020kB inactive_anon:31380kB active_file:4kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:647684kB slab_reclaimable:74208kB slab_unreclaimable:218580kB kernel_stack:2192kB pagetables:712kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [314604.966952] lowmem_reserve[]: 0 0 0 0 [314604.968311] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314604.972053] Node 0 DMA32: 22*4kB (UE) 20*8kB (EM) 2*16kB (UM) 3*32kB (UE) 1*64kB (M) 2*128kB (UE) 3*256kB (UEM) 2*512kB (UM) 3*1024kB (UEM) 1*2048kB (M) 23*4096kB (M) = 101816kB [314604.976340] Node 0 Normal: 1607*4kB (UEM) 1524*8kB (UEM) 1032*16kB (UEM) 695*32kB (UEM) 413*64kB (UEM) 212*128kB (UEM) 20*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 116060kB [314604.980613] Node 1 Normal: 1633*4kB (UEM) 2015*8kB (UEM) 1416*16kB (UM) 1097*32kB (UEM) 771*64kB (UM) 16*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 131804kB [314604.984804] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314604.986938] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314604.989089] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314604.991331] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314604.993532] 172227 total pagecache pages [314604.994984] 0 pages in swap cache [314604.996364] Swap cache stats: add 0, delete 0, find 0/0 [314604.997993] Free swap = 0kB [314604.999232] Total swap = 0kB [314605.000517] 13055868 pages RAM [314605.001838] 0 pages HighMem/MovableOnly [314605.003202] 323772 pages reserved [314605.004476] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314605.006467] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314605.008584] [ 801] 0 801 13883 110 29 0 -1000 auditd [314605.010597] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314605.012689] [ 2202] 0 2202 28244 257 57 0 -1000 sshd [314605.014679] [ 2208] 0 2208 27552 31 11 0 0 agetty [314605.016675] [ 2210] 0 2210 27552 31 8 0 0 agetty [314605.018659] [11788] 0 11788 1169 14 7 0 0 systemd-cgroups [314605.020826] [11792] 0 11792 1168 12 7 0 0 systemd-cgroups [314605.022923] [11793] 0 11793 1168 13 7 0 0 systemd-cgroups [314605.024990] [11794] 0 11794 1168 11 7 0 0 systemd-cgroups [314605.027035] [11795] 0 11795 628 12 6 0 0 systemd-cgroups [314605.029063] [11796] 0 11796 628 8 6 0 0 systemd-cgroups [314605.031102] [11798] 0 11798 1168 12 7 0 0 systemd-cgroups [314605.033161] [11799] 0 11799 1168 13 7 0 0 systemd-cgroups [314605.035142] [11800] 0 11800 1168 13 7 0 0 systemd-cgroups [314605.037101] [11801] 0 11801 82 1 5 0 0 systemd-cgroups [314605.039037] [11802] 0 11802 43 1 2 0 0 systemd-cgroups [314605.040960] Out of memory: Kill process 2208 (agetty) score 0 or sacrifice child [314605.042663] Killed process 2208 (agetty), UID 0, total-vm:110208kB, anon-rss:124kB, file-rss:0kB, shmem-rss:0kB [314605.910064] systemd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [314605.913381] systemd cpuset=/ mems_allowed=0-1 [314605.914657] CPU: 3 PID: 1 Comm: systemd Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314605.916987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314605.919920] Call Trace: [314605.920847] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314605.922193] [<ffffffff95b81668>] dump_header+0x90/0x229 [314605.923538] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314605.924921] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314605.926323] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314605.927767] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314605.929094] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314605.930532] [<ffffffff956193b8>] alloc_pages_current+0x98/0x110 [314605.931895] [<ffffffff955be057>] __page_cache_alloc+0x97/0xb0 [314605.933230] [<ffffffff955c1000>] filemap_fault+0x270/0x420 [314605.934534] [<ffffffffc05fc91e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [314605.935974] [<ffffffffc05fcb1c>] xfs_filemap_fault+0x2c/0x30 [xfs] [314605.937370] [<ffffffff955ee7da>] __do_fault.isra.61+0x8a/0x100 [314605.938712] [<ffffffff955eed8c>] do_read_fault.isra.63+0x4c/0x1b0 [314605.940096] [<ffffffff955f65d0>] handle_mm_fault+0xa20/0xfb0 [314605.941413] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314605.942726] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314605.944083] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314605.945424] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314605.946720] Mem-Info: [314605.947491] active_anon:159030 inactive_anon:14403 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40570 slab_unreclaimable:111003 mapped:8 shmem:172236 pagetables:251 bounce:0 free:91325 free_pcp:302 free_cma:0 [314605.954896] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314605.963070] lowmem_reserve[]: 0 2607 24634 24634 [314605.964382] Node 0 DMA32 free:101816kB min:13744kB low:17180kB high:20616kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314605.973163] lowmem_reserve[]: 0 0 22026 22026 [314605.974500] Node 0 Normal free:115604kB min:116096kB low:145120kB high:174144kB active_anon:17160kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:41260kB slab_reclaimable:81588kB slab_unreclaimable:199080kB kernel_stack:1728kB pagetables:312kB unstable:0kB bounce:0kB free_pcp:1208kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:136 all_unreclaimable? yes [314605.984610] lowmem_reserve[]: 0 0 0 0 [314605.985882] Node 1 Normal free:132068kB min:132216kB low:165268kB high:198324kB active_anon:618960kB inactive_anon:31380kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:647684kB slab_reclaimable:74208kB slab_unreclaimable:218644kB kernel_stack:2208kB pagetables:692kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314605.996036] lowmem_reserve[]: 0 0 0 0 [314605.997364] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314606.001046] Node 0 DMA32: 22*4kB (UE) 20*8kB (EM) 2*16kB (UM) 3*32kB (UE) 1*64kB (M) 2*128kB (UE) 3*256kB (UEM) 2*512kB (UM) 3*1024kB (UEM) 1*2048kB (M) 23*4096kB (M) = 101816kB [314606.005313] Node 0 Normal: 1353*4kB (EM) 1507*8kB (UEM) 1029*16kB (UEM) 691*32kB (UEM) 413*64kB (UEM) 205*128kB (EM) 20*256kB (M) 2*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 114860kB [314606.009590] Node 1 Normal: 1687*4kB (UEM) 2019*8kB (UEM) 1417*16kB (UM) 1097*32kB (UEM) 771*64kB (UM) 16*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 132068kB [314606.013762] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314606.015907] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314606.017993] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314606.020288] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314606.022370] 172252 total pagecache pages [314606.023744] 0 pages in swap cache [314606.025028] Swap cache stats: add 0, delete 0, find 0/0 [314606.026648] Free swap = 0kB [314606.027862] Total swap = 0kB [314606.029086] 13055868 pages RAM [314606.030331] 0 pages HighMem/MovableOnly [314606.031691] 323772 pages reserved [314606.032968] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314606.034961] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314606.037079] [ 801] 0 801 13883 110 29 0 -1000 auditd [314606.039097] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314606.041177] [ 2202] 0 2202 28244 257 57 0 -1000 sshd [314606.043166] [ 2210] 0 2210 27552 31 8 0 0 agetty [314606.045182] [11788] 0 11788 2143 14 8 0 0 systemd-cgroups [314606.047290] [11792] 0 11792 1168 12 7 0 0 systemd-cgroups [314606.049208] [11793] 0 11793 1168 13 7 0 0 systemd-cgroups [314606.050922] [11794] 0 11794 1168 11 7 0 0 systemd-cgroups [314606.052619] [11795] 0 11795 628 12 6 0 0 systemd-cgroups [314606.054302] [11796] 0 11796 628 11 6 0 0 systemd-cgroups [314606.055965] [11798] 0 11798 1168 12 7 0 0 systemd-cgroups [314606.057612] [11799] 0 11799 1168 13 7 0 0 systemd-cgroups [314606.059239] [11800] 0 11800 1168 13 7 0 0 systemd-cgroups [314606.060840] [11801] 0 11801 82 1 5 0 0 systemd-cgroups [314606.062438] [11802] 0 11802 82 1 4 0 0 systemd-cgroups [314606.064014] [11804] 0 11804 79 1 2 0 0 systemd-cgroups [314606.065571] [11805] 0 11805 1 1 1 0 0 systemd-cgroups [314606.067114] Out of memory: Kill process 2210 (agetty) score 0 or sacrifice child [314606.068479] Killed process 2210 (agetty), UID 0, total-vm:110208kB, anon-rss:124kB, file-rss:0kB, shmem-rss:0kB [314606.071230] kthreadd invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0 [314606.072949] kthreadd cpuset=/ mems_allowed=0-1 [314606.074178] CPU: 1 PID: 2 Comm: kthreadd Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314606.076946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314606.079823] Call Trace: [314606.080731] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314606.082047] [<ffffffff95b81668>] dump_header+0x90/0x229 [314606.083376] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314606.084737] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314606.086075] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314606.087429] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314606.088828] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314606.090123] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314606.091536] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314606.092910] [<ffffffff954989bd>] copy_process+0x1dd/0x1a80 [314606.094204] [<ffffffff954e283c>] ? set_next_entity+0x3c/0xe0 [314606.095568] [<ffffffff9549a411>] do_fork+0x91/0x330 [314606.096836] [<ffffffff95b8c070>] ? __schedule+0x320/0x680 [314606.098115] [<ffffffff9549a6d6>] kernel_thread+0x26/0x30 [314606.099374] [<ffffffff954c6971>] kthreadd+0x2e1/0x320 [314606.100590] [<ffffffff954c6690>] ? kthread_create_on_cpu+0x60/0x60 [314606.101983] [<ffffffff95b99ddd>] ret_from_fork_nospec_begin+0x7/0x21 [314606.103413] [<ffffffff954c6690>] ? kthread_create_on_cpu+0x60/0x60 [314606.104815] Mem-Info: [314606.105600] active_anon:159030 inactive_anon:14403 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40570 slab_unreclaimable:111003 mapped:8 shmem:172236 pagetables:251 bounce:0 free:91325 free_pcp:325 free_cma:0 [314606.113113] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314606.121332] lowmem_reserve[]: 0 2607 24634 24634 [314606.122641] Node 0 DMA32 free:101816kB min:13744kB low:17180kB high:20616kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314606.131474] lowmem_reserve[]: 0 0 22026 22026 [314606.132813] Node 0 Normal free:115604kB min:116096kB low:145120kB high:174144kB active_anon:17160kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:41260kB slab_reclaimable:81588kB slab_unreclaimable:199080kB kernel_stack:1728kB pagetables:312kB unstable:0kB bounce:0kB free_pcp:1296kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:246 all_unreclaimable? yes [314606.143009] lowmem_reserve[]: 0 0 0 0 [314606.144318] Node 1 Normal free:132068kB min:132216kB low:165268kB high:198324kB active_anon:618960kB inactive_anon:31380kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:647684kB slab_reclaimable:74208kB slab_unreclaimable:218644kB kernel_stack:2208kB pagetables:692kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314606.154541] lowmem_reserve[]: 0 0 0 0 [314606.155835] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314606.159531] Node 0 DMA32: 22*4kB (UE) 20*8kB (EM) 2*16kB (UM) 3*32kB (UE) 1*64kB (M) 2*128kB (UE) 3*256kB (UEM) 2*512kB (UM) 3*1024kB (UEM) 1*2048kB (M) 23*4096kB (M) = 101816kB [314606.163989] Node 0 Normal: 1394*4kB (UEM) 1507*8kB (UEM) 1029*16kB (UEM) 691*32kB (UEM) 413*64kB (UEM) 206*128kB (UEM) 21*256kB (UM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 114896kB [314606.168419] Node 1 Normal: 1687*4kB (UEM) 2020*8kB (UEM) 1417*16kB (UM) 1097*32kB (UEM) 771*64kB (UM) 16*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 132076kB [314606.172687] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314606.174815] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314606.176912] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314606.179039] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314606.182586] 172143 total pagecache pages [314606.183979] 0 pages in swap cache [314606.185362] Swap cache stats: add 0, delete 0, find 0/0 [314606.187047] Free swap = 0kB [314606.188346] Total swap = 0kB [314606.189645] 13055868 pages RAM [314606.190956] 0 pages HighMem/MovableOnly [314606.192399] 323772 pages reserved [314606.193741] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314606.195841] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314606.198085] [ 801] 0 801 13883 110 29 0 -1000 auditd [314606.200212] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314606.202417] [ 2202] 0 2202 28244 257 57 0 -1000 sshd [314606.204528] [11788] 0 11788 2143 18 8 0 0 systemd-cgroups [314606.206755] [11792] 0 11792 1168 12 7 0 0 systemd-cgroups [314606.208863] [11793] 0 11793 1168 13 7 0 0 systemd-cgroups [314606.211040] [11794] 0 11794 1168 11 7 0 0 systemd-cgroups [314606.213166] [11795] 0 11795 628 12 6 0 0 systemd-cgroups [314606.215347] [11796] 0 11796 628 11 6 0 0 systemd-cgroups [314606.217484] [11798] 0 11798 1168 12 7 0 0 systemd-cgroups [314606.219598] [11799] 0 11799 1168 13 7 0 0 systemd-cgroups [314606.221695] [11800] 0 11800 1168 13 7 0 0 systemd-cgroups [314606.223765] [11801] 0 11801 82 1 5 0 0 systemd-cgroups [314606.225876] [11802] 0 11802 82 1 4 0 0 systemd-cgroups [314606.227950] [11804] 0 11804 79 1 3 0 0 systemd-cgroups [314606.229995] [11805] 0 11805 79 1 3 0 0 systemd-cgroups [314606.232035] [11806] 0 11806 43 1 2 0 0 systemd-cgroups [314606.234044] [11807] 0 11807 43 1 2 0 0 systemd-cgroups [314606.236027] Out of memory: Kill process 11788 (systemd-cgroups) score 0 or sacrifice child [314606.237919] Killed process 11788 (systemd-cgroups), UID 0, total-vm:8572kB, anon-rss:64kB, file-rss:8kB, shmem-rss:0kB [314606.241312] systemd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [314606.242801] systemd cpuset=/ mems_allowed=0-1 [314606.243862] CPU: 2 PID: 1 Comm: systemd Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314606.245743] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314606.248205] Call Trace: [314606.249009] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314606.250147] [<ffffffff95b81668>] dump_header+0x90/0x229 [314606.251278] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314606.252437] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314606.253563] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314606.254701] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314606.255860] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314606.256946] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314606.258092] [<ffffffff956193b8>] alloc_pages_current+0x98/0x110 [314606.259188] [<ffffffff955be057>] __page_cache_alloc+0x97/0xb0 [314606.260306] [<ffffffff955c1000>] filemap_fault+0x270/0x420 [314606.261407] [<ffffffffc05fc91e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [314606.262810] [<ffffffffc05fcb1c>] xfs_filemap_fault+0x2c/0x30 [xfs] [314606.265195] [<ffffffff955ee7da>] __do_fault.isra.61+0x8a/0x100 [314606.266574] [<ffffffff955eed8c>] do_read_fault.isra.63+0x4c/0x1b0 [314606.268004] [<ffffffff955f65d0>] handle_mm_fault+0xa20/0xfb0 [314606.269307] [<ffffffff954d4bee>] ? finish_task_switch+0x4e/0x1c0 [314606.270655] [<ffffffff95b94653>] __do_page_fault+0x213/0x500 [314606.271956] [<ffffffff95b94a26>] trace_do_page_fault+0x56/0x150 [314606.273302] [<ffffffff95b93fa2>] do_async_page_fault+0x22/0xf0 [314606.274667] [<ffffffff95b907a8>] async_page_fault+0x28/0x30 [314606.275983] Mem-Info: [314606.276730] active_anon:159030 inactive_anon:14403 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40570 slab_unreclaimable:111083 mapped:8 shmem:172236 pagetables:251 bounce:0 free:91270 free_pcp:1201 free_cma:0 [314606.284169] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314606.292550] lowmem_reserve[]: 0 2607 24634 24634 [314606.293903] Node 0 DMA32 free:101712kB min:13744kB low:17180kB high:20616kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:1640kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:440 all_unreclaimable? yes [314606.303070] lowmem_reserve[]: 0 0 22026 22026 [314606.304439] Node 0 Normal free:115604kB min:116096kB low:145120kB high:174144kB active_anon:17160kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:41260kB slab_reclaimable:81588kB slab_unreclaimable:199080kB kernel_stack:1728kB pagetables:312kB unstable:0kB bounce:0kB free_pcp:1404kB local_pcp:396kB free_cma:0kB writeback_tmp:0kB pages_scanned:429 all_unreclaimable? yes [314606.314531] lowmem_reserve[]: 0 0 0 0 [314606.315783] Node 1 Normal free:131952kB min:132216kB low:165268kB high:198324kB active_anon:618960kB inactive_anon:31380kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:647684kB slab_reclaimable:74208kB slab_unreclaimable:218964kB kernel_stack:2208kB pagetables:692kB unstable:0kB bounce:0kB free_pcp:1640kB local_pcp:140kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314606.326483] lowmem_reserve[]: 0 0 0 0 [314606.327818] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314606.331531] Node 0 DMA32: 23*4kB (EM) 23*8kB (UEM) 3*16kB (UM) 4*32kB (EM) 2*64kB (UM) 2*128kB (UE) 2*256kB (UE) 1*512kB (U) 3*1024kB (UEM) 1*2048kB (M) 23*4096kB (M) = 101188kB [314606.335761] Node 0 Normal: 1427*4kB (UEM) 1509*8kB (UEM) 1031*16kB (UEM) 693*32kB (UEM) 413*64kB (UEM) 206*128kB (UEM) 21*256kB (UM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 115140kB [314606.340088] Node 1 Normal: 1320*4kB (UEM) 1977*8kB (UEM) 1403*16kB (UM) 1075*32kB (EM) 764*64kB (M) 17*128kB (UM) 1*256kB (U) 4*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 131320kB [314606.344369] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314606.346503] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314606.348591] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314606.350711] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314606.352785] 172143 total pagecache pages [314606.354163] 0 pages in swap cache [314606.355447] Swap cache stats: add 0, delete 0, find 0/0 [314606.357024] Free swap = 0kB [314606.358245] Total swap = 0kB [314606.359459] 13055868 pages RAM [314606.360695] 0 pages HighMem/MovableOnly [314606.362046] 323772 pages reserved [314606.363324] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314606.365311] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314606.367421] [ 801] 0 801 13883 110 29 0 -1000 auditd [314606.369427] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314606.371508] [ 2202] 0 2202 28244 257 57 0 -1000 sshd [314606.373500] [11792] 0 11792 1168 12 7 0 0 systemd-cgroups [314606.375614] [11793] 0 11793 1168 13 7 0 0 systemd-cgroups [314606.377709] [11794] 0 11794 1168 11 7 0 0 systemd-cgroups [314606.379786] [11795] 0 11795 628 12 6 0 0 systemd-cgroups [314606.381864] [11796] 0 11796 628 11 6 0 0 systemd-cgroups [314606.383908] [11798] 0 11798 1168 13 7 0 0 systemd-cgroups [314606.385947] [11799] 0 11799 1168 13 7 0 0 systemd-cgroups [314606.387960] [11800] 0 11800 1168 13 7 0 0 systemd-cgroups [314606.389980] [11801] 0 11801 82 1 5 0 0 systemd-cgroups [314606.391967] [11802] 0 11802 82 1 4 0 0 systemd-cgroups [314606.393922] [11804] 0 11804 82 1 4 0 0 systemd-cgroups [314606.395874] [11805] 0 11805 82 1 3 0 0 systemd-cgroups [314606.397804] [11806] 0 11806 79 1 3 0 0 systemd-cgroups [314606.399722] [11807] 0 11807 82 1 4 0 0 systemd-cgroups [314606.401610] Out of memory: Kill process 11793 (systemd-cgroups) score 0 or sacrifice child [314606.403420] Killed process 11793 (systemd-cgroups), UID 0, total-vm:4672kB, anon-rss:52kB, file-rss:0kB, shmem-rss:0kB [314606.406334] kiblnd_sd_01_01 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [314606.408087] kiblnd_sd_01_01 cpuset=/ mems_allowed=1 [314606.409544] CPU: 7 PID: 17024 Comm: kiblnd_sd_01_01 Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314606.412550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314606.415486] Call Trace: [314606.416408] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314606.417740] [<ffffffff95b81668>] dump_header+0x90/0x229 [314606.419088] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314606.420472] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314606.421824] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314606.423210] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314606.424630] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314606.425979] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314606.427408] [<ffffffff95606d5f>] __vmalloc_node_range+0x12f/0x280 [314606.428817] [<ffffffffc092181d>] ? kiblnd_create_tx_pool+0x57d/0x14e0 [ko2iblnd] [314606.430426] [<ffffffff95606f4e>] vzalloc_node+0x4e/0x50 [314606.431697] [<ffffffffc092181d>] ? kiblnd_create_tx_pool+0x57d/0x14e0 [ko2iblnd] [314606.433304] [<ffffffffc092181d>] kiblnd_create_tx_pool+0x57d/0x14e0 [ko2iblnd] [314606.434888] [<ffffffffc0923065>] kiblnd_pool_alloc_node+0x275/0x3e0 [ko2iblnd] [314606.436477] [<ffffffffc0929f4b>] kiblnd_get_idle_tx+0x2b/0x270 [ko2iblnd] [314606.437986] [<ffffffffc092f3b6>] kiblnd_send+0x276/0xa20 [ko2iblnd] [314606.439477] [<ffffffffc09fbefd>] lnet_ni_send+0x4d/0xe0 [lnet] [314606.440853] [<ffffffffc09ff2d2>] lnet_send+0x82/0x1c0 [lnet] [314606.442207] [<ffffffffc09f0b09>] lnet_finalize+0x479/0xd50 [lnet] [314606.443625] [<ffffffffc093034d>] kiblnd_recv+0x1cd/0x7c0 [ko2iblnd] [314606.445065] [<ffffffffc09fb5d8>] lnet_ni_recv+0xc8/0x330 [lnet] [314606.446445] [<ffffffffc0a0453f>] lnet_parse+0xabf/0x11e0 [lnet] [314606.447844] [<ffffffffc05507fb>] ? mlx5_ib_poll_cq+0x40b/0xdd0 [mlx5_ib] [314606.449346] [<ffffffffc0930cf3>] kiblnd_handle_rx+0x213/0x6b0 [ko2iblnd] [314606.450849] [<ffffffffc0937ec2>] kiblnd_scheduler+0xfa2/0x11c0 [ko2iblnd] [314606.452371] [<ffffffff954db2e0>] ? wake_up_state+0x20/0x20 [314606.453686] [<ffffffffc0936f20>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [314606.455197] [<ffffffff954c5f91>] kthread+0xd1/0xe0 [314606.456478] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314606.457912] [<ffffffff95b99ddd>] ret_from_fork_nospec_begin+0x7/0x21 [314606.459391] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314606.460814] Mem-Info: [314606.461649] active_anon:159030 inactive_anon:14403 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40570 slab_unreclaimable:111083 mapped:8 shmem:172236 pagetables:251 bounce:0 free:91240 free_pcp:445 free_cma:0 [314606.469575] Node 1 Normal free:131952kB min:132216kB low:165268kB high:198324kB active_anon:618960kB inactive_anon:31380kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:647684kB slab_reclaimable:74208kB slab_unreclaimable:218964kB kernel_stack:2208kB pagetables:692kB unstable:0kB bounce:0kB free_pcp:1780kB local_pcp:116kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314606.479228] lowmem_reserve[]: 0 0 0 0 [314606.480771] Node 1 Normal: 1320*4kB (UEM) 1978*8kB (UEM) 1404*16kB (UM) 1075*32kB (EM) 764*64kB (M) 17*128kB (UM) 1*256kB (U) 4*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 131344kB [314606.484592] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314606.486498] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314606.488357] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314606.490247] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314606.492097] 172143 total pagecache pages [314606.493235] 0 pages in swap cache [314606.494277] Swap cache stats: add 0, delete 0, find 0/0 [314606.495623] Free swap = 0kB [314606.496598] Total swap = 0kB [314606.497567] 13055868 pages RAM [314606.498562] 0 pages HighMem/MovableOnly [314606.499677] 323772 pages reserved [314606.500703] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314606.502458] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314606.504334] [ 801] 0 801 13883 110 29 0 -1000 auditd [314606.506113] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314606.507955] [ 2202] 0 2202 28244 257 57 0 -1000 sshd [314606.509710] [11792] 0 11792 1168 12 7 0 0 systemd-cgroups [314606.511604] [11794] 0 11794 1168 11 7 0 0 systemd-cgroups [314606.513497] [11795] 0 11795 628 12 6 0 0 systemd-cgroups [314606.515389] [11796] 0 11796 628 11 6 0 0 systemd-cgroups [314606.517271] [11798] 0 11798 1168 13 7 0 0 systemd-cgroups [314606.519145] [11799] 0 11799 1168 13 7 0 0 systemd-cgroups [314606.521020] [11800] 0 11800 1168 13 7 0 0 systemd-cgroups [314606.522895] [11801] 0 11801 82 1 5 0 0 systemd-cgroups [314606.524760] [11802] 0 11802 82 1 4 0 0 systemd-cgroups [314606.526622] [11804] 0 11804 82 1 4 0 0 systemd-cgroups [314606.528472] [11805] 0 11805 82 1 3 0 0 systemd-cgroups [314606.530325] [11806] 0 11806 79 1 3 0 0 systemd-cgroups [314606.532164] [11807] 0 11807 82 1 4 0 0 systemd-cgroups [314606.533991] Out of memory: Kill process 11798 (systemd-cgroups) score 0 or sacrifice child [314606.535754] Killed process 11798 (systemd-cgroups), UID 0, total-vm:4672kB, anon-rss:52kB, file-rss:0kB, shmem-rss:0kB [314606.538380] kiblnd_sd_01_01 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [314606.539776] kiblnd_sd_01_01 cpuset=/ mems_allowed=1 [314606.540807] CPU: 5 PID: 17024 Comm: kiblnd_sd_01_01 Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314606.543148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314606.545501] Call Trace: [314606.546295] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314606.547387] [<ffffffff95b81668>] dump_header+0x90/0x229 [314606.548474] [<ffffffff95506a42>] ? ktime_get_ts64+0x52/0xf0 [314606.549601] [<ffffffff9555e0ff>] ? delayacct_end+0x8f/0xb0 [314606.550691] [<ffffffff955c25ad>] oom_kill_process+0x2cd/0x490 [314606.551790] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314606.552956] [<ffffffff955c2c9a>] out_of_memory+0x31a/0x500 [314606.554005] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314606.556235] [<ffffffff95606d5f>] __vmalloc_node_range+0x12f/0x280 [314606.558411] [<ffffffffc092181d>] ? kiblnd_create_tx_pool+0x57d/0x14e0 [ko2iblnd] [314606.560769] [<ffffffff95606f4e>] vzalloc_node+0x4e/0x50 [314606.561808] [<ffffffffc092181d>] ? kiblnd_create_tx_pool+0x57d/0x14e0 [ko2iblnd] [314606.563112] [<ffffffffc092181d>] kiblnd_create_tx_pool+0x57d/0x14e0 [ko2iblnd] [314606.564353] [<ffffffffc0923065>] kiblnd_pool_alloc_node+0x275/0x3e0 [ko2iblnd] [314606.565633] [<ffffffffc0929f4b>] kiblnd_get_idle_tx+0x2b/0x270 [ko2iblnd] [314606.566847] [<ffffffffc092f3b6>] kiblnd_send+0x276/0xa20 [ko2iblnd] [314606.569070] [<ffffffffc09fbefd>] lnet_ni_send+0x4d/0xe0 [lnet] [314606.570252] [<ffffffffc09ff2d2>] lnet_send+0x82/0x1c0 [lnet] [314606.571435] [<ffffffffc09f0b09>] lnet_finalize+0x479/0xd50 [lnet] [314606.572672] [<ffffffffc093034d>] kiblnd_recv+0x1cd/0x7c0 [ko2iblnd] [314606.573935] [<ffffffffc09fb5d8>] lnet_ni_recv+0xc8/0x330 [lnet] [314606.575157] [<ffffffffc0a0453f>] lnet_parse+0xabf/0x11e0 [lnet] [314606.576384] [<ffffffffc05507fb>] ? mlx5_ib_poll_cq+0x40b/0xdd0 [mlx5_ib] [314606.577690] [<ffffffffc0930cf3>] kiblnd_handle_rx+0x213/0x6b0 [ko2iblnd] [314606.579011] [<ffffffffc0937ec2>] kiblnd_scheduler+0xfa2/0x11c0 [ko2iblnd] [314606.582734] [<ffffffff954db2e0>] ? wake_up_state+0x20/0x20 [314606.583919] [<ffffffffc0936f20>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [314606.585238] [<ffffffff954c5f91>] kthread+0xd1/0xe0 [314606.586338] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314606.587580] [<ffffffff95b99ddd>] ret_from_fork_nospec_begin+0x7/0x21 [314606.588872] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314606.590131] Mem-Info: [314606.590907] active_anon:159030 inactive_anon:14403 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40570 slab_unreclaimable:111083 mapped:8 shmem:172236 pagetables:251 bounce:0 free:91240 free_pcp:456 free_cma:0 [314606.597911] Node 1 Normal free:131952kB min:132216kB low:165268kB high:198324kB active_anon:618960kB inactive_anon:31380kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:26112000kB managed:25686612kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:647684kB slab_reclaimable:74208kB slab_unreclaimable:218964kB kernel_stack:2208kB pagetables:692kB unstable:0kB bounce:0kB free_pcp:1824kB local_pcp:304kB free_cma:0kB writeback_tmp:0kB pages_scanned:6 all_unreclaimable? yes [314606.606117] lowmem_reserve[]: 0 0 0 0 [314606.607181] Node 1 Normal: 1320*4kB (UEM) 1979*8kB (UEM) 1404*16kB (UM) 1075*32kB (EM) 764*64kB (M) 17*128kB (UM) 1*256kB (U) 4*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 131352kB [314606.610487] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314606.612096] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314606.613659] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314606.615273] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314606.616848] 172143 total pagecache pages [314606.617875] 0 pages in swap cache [314606.618867] Swap cache stats: add 0, delete 0, find 0/0 [314606.620099] Free swap = 0kB [314606.621008] Total swap = 0kB [314606.621905] 13055868 pages RAM [314606.622810] 0 pages HighMem/MovableOnly [314606.623810] 323772 pages reserved [314606.624734] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314606.626265] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314606.627866] [ 801] 0 801 13883 110 29 0 -1000 auditd [314606.629384] [ 878] 81 878 14536 162 32 0 -900 dbus-daemon [314606.630952] [ 2202] 0 2202 28244 257 57 0 -1000 sshd [314606.632449] [11792] 0 11792 1168 12 7 0 0 systemd-cgroups [314606.634056] [11794] 0 11794 1168 11 7 0 0 systemd-cgroups [314606.635659] [11795] 0 11795 628 12 6 0 0 systemd-cgroups [314606.637263] [11796] 0 11796 628 11 6 0 0 systemd-cgroups [314606.638856] [11799] 0 11799 1168 13 7 0 0 systemd-cgroups [314606.640438] [11800] 0 11800 1168 13 7 0 0 systemd-cgroups [314606.642016] [11801] 0 11801 82 1 5 0 0 systemd-cgroups [314606.643587] [11802] 0 11802 82 1 4 0 0 systemd-cgroups [314606.645160] [11804] 0 11804 82 1 4 0 0 systemd-cgroups [314606.646726] [11805] 0 11805 82 1 3 0 0 systemd-cgroups [314606.648292] [11806] 0 11806 79 1 3 0 0 systemd-cgroups [314606.649841] [11807] 0 11807 82 1 4 0 0 systemd-cgroups [314606.651386] Out of memory: Kill process 11799 (systemd-cgroups) score 0 or sacrifice child [314606.652872] Killed process 11799 (systemd-cgroups), UID 0, total-vm:4672kB, anon-rss:52kB, file-rss:0kB, shmem-rss:0kB [314606.656254] Out of memory: Kill process 11800 (systemd-cgroups) score 0 or sacrifice child [314606.657840] Killed process 11800 (systemd-cgroups), UID 0, total-vm:4672kB, anon-rss:52kB, file-rss:0kB, shmem-rss:0kB [314606.661313] Out of memory: Kill process 11792 (systemd-cgroups) score 0 or sacrifice child [314606.662798] Killed process 11792 (systemd-cgroups), UID 0, total-vm:4672kB, anon-rss:48kB, file-rss:0kB, shmem-rss:0kB [314606.665238] Out of memory: Kill process 11794 (systemd-cgroups) score 0 or sacrifice child [314606.667078] Killed process 11794 (systemd-cgroups), UID 0, total-vm:4672kB, anon-rss:44kB, file-rss:0kB, shmem-rss:0kB [314606.670237] Out of memory: Kill process 11795 (systemd-cgroups) score 0 or sacrifice child [314606.671760] Killed process 11795 (systemd-cgroups), UID 0, total-vm:2512kB, anon-rss:48kB, file-rss:0kB, shmem-rss:0kB [314606.674236] Out of memory: Kill process 11796 (systemd-cgroups) score 0 or sacrifice child [314606.676090] Killed process 11796 (systemd-cgroups), UID 0, total-vm:2512kB, anon-rss:44kB, file-rss:0kB, shmem-rss:0kB [314606.679229] Out of memory: Kill process 11801 (systemd-cgroups) score 0 or sacrifice child [314606.680730] Killed process 11801 (systemd-cgroups), UID 0, total-vm:328kB, anon-rss:4kB, file-rss:0kB, shmem-rss:0kB [314606.683222] Out of memory: Kill process 11802 (systemd-cgroups) score 0 or sacrifice child [314606.685064] Killed process 11802 (systemd-cgroups), UID 0, total-vm:328kB, anon-rss:4kB, file-rss:0kB, shmem-rss:0kB [314606.688210] Out of memory: Kill process 11804 (systemd-cgroups) score 0 or sacrifice child [314606.689722] Killed process 11804 (systemd-cgroups), UID 0, total-vm:328kB, anon-rss:4kB, file-rss:0kB, shmem-rss:0kB [314606.692199] Out of memory: Kill process 11807 (systemd-cgroups) score 0 or sacrifice child [314606.693697] Killed process 11807 (systemd-cgroups), UID 0, total-vm:328kB, anon-rss:4kB, file-rss:0kB, shmem-rss:0kB [314606.696200] Out of memory: Kill process 11805 (systemd-cgroups) score 0 or sacrifice child [314606.698066] Killed process 11805 (systemd-cgroups), UID 0, total-vm:328kB, anon-rss:4kB, file-rss:0kB, shmem-rss:0kB [314606.701203] Out of memory: Kill process 11806 (systemd-cgroups) score 0 or sacrifice child [314606.702749] Killed process 11806 (systemd-cgroups), UID 0, total-vm:316kB, anon-rss:4kB, file-rss:0kB, shmem-rss:0kB [314606.705198] Out of memory: Kill process 878 (dbus-daemon) score 0 or sacrifice child [314606.707036] Killed process 878 (dbus-daemon), UID 81, total-vm:58144kB, anon-rss:648kB, file-rss:0kB, shmem-rss:0kB [314606.711517] Out of memory: Kill process 11809 (systemd-cgroups) score 0 or sacrifice child [314606.713123] Killed process 11809 (systemd-cgroups), UID 0, total-vm:328kB, anon-rss:4kB, file-rss:0kB, shmem-rss:0kB [314606.716229] kiblnd_sd_00_01 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [314606.717839] kiblnd_sd_00_01 cpuset=/ mems_allowed=0 [314606.719063] CPU: 1 PID: 17022 Comm: kiblnd_sd_00_01 Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314606.721784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314606.724529] Call Trace: [314606.725521] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314606.726821] [<ffffffff95b81668>] dump_header+0x90/0x229 [314606.728152] [<ffffffff95533671>] ? cpuset_mems_allowed_intersects+0x21/0x30 [314606.729672] [<ffffffff955c1f9d>] ? oom_unkillable_task+0xcd/0x120 [314606.731118] [<ffffffff955c2046>] ? find_lock_task_mm+0x56/0xc0 [314606.732521] [<ffffffff955c2e6e>] out_of_memory+0x4ee/0x500 [314606.733890] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314606.735357] [<ffffffff95606d5f>] __vmalloc_node_range+0x12f/0x280 [314606.736812] [<ffffffffc092181d>] ? kiblnd_create_tx_pool+0x57d/0x14e0 [ko2iblnd] [314606.738429] [<ffffffff95606f4e>] vzalloc_node+0x4e/0x50 [314606.739795] [<ffffffffc092181d>] ? kiblnd_create_tx_pool+0x57d/0x14e0 [ko2iblnd] [314606.741445] [<ffffffffc092181d>] kiblnd_create_tx_pool+0x57d/0x14e0 [ko2iblnd] [314606.743059] [<ffffffffc0923065>] kiblnd_pool_alloc_node+0x275/0x3e0 [ko2iblnd] [314606.744662] [<ffffffffc0929f4b>] kiblnd_get_idle_tx+0x2b/0x270 [ko2iblnd] [314606.746212] [<ffffffffc092f3b6>] kiblnd_send+0x276/0xa20 [ko2iblnd] [314606.747700] [<ffffffffc09fbefd>] lnet_ni_send+0x4d/0xe0 [lnet] [314606.749114] [<ffffffffc09ff2d2>] lnet_send+0x82/0x1c0 [lnet] [314606.750490] [<ffffffffc09f0b09>] lnet_finalize+0x479/0xd50 [lnet] [314606.751909] [<ffffffffc092c2bd>] kiblnd_tx_done+0x10d/0x3c0 [ko2iblnd] [314606.753390] [<ffffffffc09377bd>] kiblnd_scheduler+0x89d/0x11c0 [ko2iblnd] [314606.754868] [<ffffffff954db2e0>] ? wake_up_state+0x20/0x20 [314606.756192] [<ffffffffc0936f20>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [314606.757623] [<ffffffff954c5f91>] kthread+0xd1/0xe0 [314606.758832] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314606.760201] [<ffffffff95b99ddd>] ret_from_fork_nospec_begin+0x7/0x21 [314606.761574] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314606.762910] Mem-Info: [314606.763783] active_anon:158884 inactive_anon:14403 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:40570 slab_unreclaimable:111083 mapped:8 shmem:172236 pagetables:251 bounce:0 free:91302 free_pcp:701 free_cma:0 [314606.770855] Node 0 DMA free:15812kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314606.777764] lowmem_reserve[]: 0 2607 24634 24634 [314606.778929] Node 0 DMA32 free:101592kB min:13744kB low:17180kB high:20616kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129208kB managed:2670424kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:6484kB slab_unreclaimable:26256kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:852kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314606.786116] lowmem_reserve[]: 0 0 22026 22026 [314606.789071] Node 0 Normal free:115728kB min:116096kB low:145120kB high:174144kB active_anon:16576kB inactive_anon:26232kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:22966272kB managed:22555440kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:41260kB slab_reclaimable:81588kB slab_unreclaimable:199080kB kernel_stack:1728kB pagetables:312kB unstable:0kB bounce:0kB free_pcp:1952kB local_pcp:112kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [314606.797155] lowmem_reserve[]: 0 0 0 0 [314606.798270] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15812kB [314606.801305] Node 0 DMA32: 27*4kB (UEM) 24*8kB (UEM) 6*16kB (UM) 8*32kB (UEM) 2*64kB (UM) 1*128kB (E) 1*256kB (E) 2*512kB (U) 3*1024kB (UEM) 1*2048kB (M) 23*4096kB (M) = 101516kB [314606.804789] Node 0 Normal: 1376*4kB (UEM) 1484*8kB (UEM) 1019*16kB (UEM) 683*32kB (EM) 405*64kB (UEM) 209*128kB (UEM) 23*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 114608kB [314606.808319] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314606.810017] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314606.811671] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [314606.813366] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [314606.815005] 172252 total pagecache pages [314606.816142] 0 pages in swap cache [314606.817204] Swap cache stats: add 0, delete 0, find 0/0 [314606.818486] Free swap = 0kB [314606.819508] Total swap = 0kB [314606.820518] 13055868 pages RAM [314606.821543] 0 pages HighMem/MovableOnly [314606.822652] 323772 pages reserved [314606.823705] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [314606.825294] [ 638] 0 638 11382 145 24 0 -1000 systemd-udevd [314606.826965] [ 801] 0 801 13883 110 29 0 -1000 auditd [314606.828568] [ 2202] 0 2202 28244 257 57 0 -1000 sshd [314606.830158] Kernel panic - not syncing: Out of memory and no killable processes... [314606.832522] CPU: 1 PID: 17022 Comm: kiblnd_sd_00_01 Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.76.1.el7.x86_64 #1 [314606.835263] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014 [314606.837984] Call Trace: [314606.838981] [<ffffffff95b865c9>] dump_stack+0x19/0x1b [314606.840289] [<ffffffff95b802d1>] panic+0xe8/0x21f [314606.841546] [<ffffffff955c2e7c>] out_of_memory+0x4fc/0x500 [314606.842880] [<ffffffff955c9894>] __alloc_pages_nodemask+0xad4/0xbe0 [314606.844307] [<ffffffff95606d5f>] __vmalloc_node_range+0x12f/0x280 [314606.845707] [<ffffffffc092181d>] ? kiblnd_create_tx_pool+0x57d/0x14e0 [ko2iblnd] [314606.847227] [<ffffffff95606f4e>] vzalloc_node+0x4e/0x50 [314606.848492] [<ffffffffc092181d>] ? kiblnd_create_tx_pool+0x57d/0x14e0 [ko2iblnd] [314606.850004] [<ffffffffc092181d>] kiblnd_create_tx_pool+0x57d/0x14e0 [ko2iblnd] [314606.851483] [<ffffffffc0923065>] kiblnd_pool_alloc_node+0x275/0x3e0 [ko2iblnd] [314606.852949] [<ffffffffc0929f4b>] kiblnd_get_idle_tx+0x2b/0x270 [ko2iblnd] [314606.854353] [<ffffffffc092f3b6>] kiblnd_send+0x276/0xa20 [ko2iblnd] [314606.855681] [<ffffffffc09fbefd>] lnet_ni_send+0x4d/0xe0 [lnet] [314606.856945] [<ffffffffc09ff2d2>] lnet_send+0x82/0x1c0 [lnet] [314606.858184] [<ffffffffc09f0b09>] lnet_finalize+0x479/0xd50 [lnet] [314606.859447] [<ffffffffc092c2bd>] kiblnd_tx_done+0x10d/0x3c0 [ko2iblnd] [314606.860758] [<ffffffffc09377bd>] kiblnd_scheduler+0x89d/0x11c0 [ko2iblnd] [314606.862074] [<ffffffff954db2e0>] ? wake_up_state+0x20/0x20 [314606.863254] [<ffffffffc0936f20>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [314606.864546] [<ffffffff954c5f91>] kthread+0xd1/0xe0 [314606.865613] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40 [314606.866806] [<ffffffff95b99ddd>] ret_from_fork_nospec_begin+0x7/0x21 [314606.868024] [<ffffffff954c5ec0>] ? insert_kthread_work+0x40/0x40