2012-01-19 19:10:42 Lustre: DEBUG MARKER: ==== Checking the clients loads AFTER failover -- failure NOT OK 2012-01-19 19:10:42 Lustre: 10187:0:(ldlm_lib.c:2112:target_queue_recovery_request()) Next recovery transno: 103084618172, current: 103084266513, replaying 2012-01-19 19:10:42 Lustre: 10187:0:(ldlm_lib.c:2112:target_queue_recovery_request()) Skipped 2 previous similar messages 2012-01-19 19:10:45 Lustre: 10187:0:(ldlm_lib.c:909:target_handle_connect()) lustre-MDT0000: connection from 48c02cc4-1fb5-a023-ee3f-f68e833b6339@192.168.115.142@o2ib1 recovering/t103084264787 exp ffff8800ccfe2400 cur 1327029045 last 1327029025 2012-01-19 19:10:45 Lustre: 10187:0:(ldlm_lib.c:909:target_handle_connect()) Skipped 4 previous similar messages 2012-01-19 19:10:45 Lustre: 10187:0:(ldlm_lib.c:2112:target_queue_recovery_request()) Next recovery transno: 103084618172, current: 103084264982, replaying 2012-01-19 19:10:45 Lustre: 10187:0:(ldlm_lib.c:2112:target_queue_recovery_request()) Skipped 1 previous similar message 2012-01-19 19:10:47 Lustre: 6099:0:(mgc_request.c:1519:mgc_process_recover_log()) Process recover log lustre-mdtir error -22 2012-01-19 19:10:50 Lustre: 10187:0:(ldlm_lib.c:909:target_handle_connect()) lustre-MDT0000: connection from c139137f-7a30-43a4-aad3-579294b0cdc2@192.168.113.116@o2ib1 recovering/t103084528544 exp ffff8800cc183800 cur 1327029049 last 1327029025 2012-01-19 19:10:50 Lustre: 10187:0:(ldlm_lib.c:909:target_handle_connect()) Skipped 70 previous similar messages 2012-01-19 19:10:50 Lustre: 10187:0:(ldlm_lib.c:2112:target_queue_recovery_request()) Next recovery transno: 103084618172, current: 103084549779, replaying 2012-01-19 19:10:50 Lustre: 10187:0:(ldlm_lib.c:2112:target_queue_recovery_request()) Skipped 140 previous similar messages 2012-01-19 19:10:57 LustreError: 10421:0:(lu_object.c:116:lu_object_put()) ASSERTION(cfs_list_empty(&top->loh_lru)) failed 2012-01-19 19:10:57 LustreError: 10421:0:(lu_object.c:116:lu_object_put()) LBUG 2012-01-19 19:10:57 Pid: 10421, commInitializing cgroup subsys cpuset 2012-01-19 19:10:57 Initializing cgroup subsys cpu 2012-01-19 19:10:57 Linux version 2.6.32-131.17.1.el6_lustre.x86_64 (jenkins@rhel6-64-build.lab.whamcloud.com) (gcc version 4.4.5 20110214 (Red Hat 4.4.5-6) (GCC) ) #1 SMP Wed Jan 11 11:18:45 PST 2012 2012-01-19 19:10:57 Command line: nmi_watchdog=0 initrd=initramfs-2.6.32-131.17.1.el6_lustre.x86_64.img console=tty0 console=ttyS0,115200n8 BOOT_IMAGE=vmlinuz-2.6.32-131.17.1.el6_lustre.x86_64 BOOTIF=01-00-22-19-57-4b-03 irqpoll maxcpus=1 reset_devices cgroup_disable=memory memmap=exactmap memmap=636K@4K memmap=195952K@33404K elfcorehdr=229356K memmap=4K$0K memmap=88K$3398244K memmap=124K#3398332K memmap=9416K$3398456K memmap=262144K$3670016K memmap=32768K$4161536K 2012-01-19 19:10:57 KERNEL supported cpus: 2012-01-19 19:10:57 Intel GenuineIntel 2012-01-19 19:10:57 AMD AuthenticAMD 2012-01-19 19:10:57 Centaur CentaurHauls 2012-01-19 19:10:57 BIOS-provided physical RAM map: 2012-01-19 19:10:57 BIOS-e820: 0000000000000100 - 00000000000a0000 (usable) 2012-01-19 19:10:57 BIOS-e820: 0000000000100000 - 00000000cf699000 (usable) 2012-01-19 19:10:57 BIOS-e820: 00000000cf699000 - 00000000cf6af000 (reserved) 2012-01-19 19:10:57 BIOS-e820: 00000000cf6af000 - 00000000cf6ce000 (ACPI data) 2012-01-19 19:10:57 BIOS-e820: 00000000cf6ce000 - 00000000d0000000 (reserved) 2012-01-19 19:10:57 BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) 2012-01-19 19:10:57 BIOS-e820: 00000000fe000000 - 0000000100000000 (reserved) 2012-01-19 19:10:57 BIOS-e820: 0000000100000000 - 0000000330000000 (usable) 2012-01-19 19:10:57 last_pfn = 0x330000 max_arch_pfn = 0x400000000 2012-01-19 19:10:57 user-defined physical RAM map: 2012-01-19 19:10:57 user: 0000000000000000 - 0000000000001000 (reserved) 2012-01-19 19:10:57 user: 0000000000001000 - 00000000000a0000 (usable) 2012-01-19 19:10:57 user: 000000000209f000 - 000000000dffb000 (usable) 2012-01-19 19:10:57 user: 00000000cf699000 - 00000000cf6af000 (reserved) 2012-01-19 19:10:57 user: 00000000cf6af000 - 00000000cf6ce000 (ACPI data) 2012-01-19 19:10:57 user: 00000000cf6ce000 - 00000000d0000000 (reserved) 2012-01-19 19:10:57 user: 00000000e0000000 - 00000000f0000000 (reserved) 2012-01-19 19:10:57 user: 00000000fe000000 - 0000000100000000 (reserved) 2012-01-19 19:10:57 DMI 2.6 present. 2012-01-19 19:10:57 SMBIOS version 2.6 @ 0xFCB00 2012-01-19 19:10:57 last_pfn = 0xdffb max_arch_pfn = 0x400000000 2012-01-19 19:10:57 x86 PAT enabled: cpu 0, old 0x7010600070106, new 0x7010600070106 2012-01-19 19:10:57 init_memory_mapping: 0000000000000000-000000000dffb000 2012-01-19 19:10:57 RAMDISK: 0d5b0000 - 0dfee76f 2012-01-19 19:10:57 ACPI: RSDP 00000000000f1640 00024 (v02 DELL ) 2012-01-19 19:10:57 ACPI: XSDT 00000000000f1744 0009C (v01 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:57 ACPI: FACP 00000000cf6c3f9c 000F4 (v03 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:57 ACPI: DSDT 00000000cf6af000 0320F (v01 DELL PE_SC3 00000001 INTL 20050624) 2012-01-19 19:10:57 ACPI: FACS 00000000cf6c6000 00040 2012-01-19 19:10:57 ACPI: APIC 00000000cf6c3478 0015E (v01 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:57 ACPI: SPCR 00000000cf6c35d8 00050 (v01 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:57 ACPI: HPET 00000000cf6c362c 00038 (v01 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:57 ACPI: DM__ 00000000cf6c3668 001A8 (v01 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:57 ACPI: MCFG 00000000cf6c38c4 0003C (v01 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:57 ACPI: WD__ 00000000cf6c3904 00134 (v01 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:58 ACPI: SLIC 00000000cf6c3a3c 00024 (v01 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:58 ACPI: ERST 00000000cf6b2390 00270 (v01 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:58 ACPI: HEST 00000000cf6b2600 0027C (v01 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:58 ACPI: BERT 00000000cf6b2210 00030 (v01 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:58 ACPI: EINJ 00000000cf6b2240 00150 (v01 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:58 ACPI: SRAT 00000000cf6c3bc0 00370 (v01 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:58 ACPI: TCPA 00000000cf6c3f34 00064 (v02 DELL PE_SC3 00000001 DELL 00000001) 2012-01-19 19:10:58 ACPI: SSDT 00000000cf6c7000 043F4 (v01 INTEL PPM RCM 80000001 INTL 20061109) 2012-01-19 19:10:58 SRAT: PXM 1 -> APIC 16 -> Node 0 2012-01-19 19:10:58 SRAT: PXM 2 -> APIC 0 -> Node 1 2012-01-19 19:10:58 SRAT: PXM 1 -> APIC 18 -> Node 0 2012-01-19 19:10:58 SRAT: PXM 2 -> APIC 2 -> Node 1 2012-01-19 19:10:58 SRAT: PXM 1 -> APIC 20 -> Node 0 2012-01-19 19:10:58 SRAT: PXM 2 -> APIC 4 -> Node 1 2012-01-19 19:10:58 SRAT: PXM 1 -> APIC 22 -> Node 0 2012-01-19 19:10:58 SRAT: PXM 2 -> APIC 6 -> Node 1 2012-01-19 19:10:58 SRAT: PXM 1 -> APIC 17 -> Node 0 2012-01-19 19:10:58 SRAT: PXM 2 -> APIC 1 -> Node 1 2012-01-19 19:10:58 SRAT: PXM 1 -> APIC 19 -> Node 0 2012-01-19 19:10:58 SRAT: PXM 2 -> APIC 3 -> Node 1 2012-01-19 19:10:58 SRAT: PXM 1 -> APIC 21 -> Node 0 2012-01-19 19:10:58 SRAT: PXM 2 -> APIC 5 -> Node 1 2012-01-19 19:10:58 SRAT: PXM 1 -> APIC 23 -> Node 0 2012-01-19 19:10:58 SRAT: PXM 2 -> APIC 7 -> Node 1 2012-01-19 19:10:58 SRAT: Node 1 PXM 2 0-d0000000 2012-01-19 19:10:58 SRAT: Node 1 PXM 2 100000000-1b0000000 2012-01-19 19:10:58 SRAT: Node 0 PXM 1 1b0000000-330000000 2012-01-19 19:10:58 Bootmem setup node 1 0000000000000000-000000000dffb000 2012-01-19 19:10:58 NODE_DATA [0000000000009040 - 000000000003d03f] 2012-01-19 19:10:58 bootmap [000000000003e000 - 000000000003fbff] pages 2 2012-01-19 19:10:58 (7 early reservations) ==> bootmem [0000000000 - 000dffb000] 2012-01-19 19:10:58 #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] 2012-01-19 19:10:58 #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] 2012-01-19 19:10:58 #2 [0003000000 - 0003f47524] TEXT DATA BSS ==> [0003000000 - 0003f47524] 2012-01-19 19:10:58 #3 [000d5b0000 - 000dfee76f] RAMDISK ==> [000d5b0000 - 000dfee76f] 2012-01-19 19:10:58 #4 [000009e400 - 0000100000] BIOS reserved ==> [000009e400 - 0000100000] 2012-01-19 19:10:58 #5 [0003f48000 - 0003f48761] BRK ==> [0003f48000 - 0003f48761] 2012-01-19 19:10:58 #6 [0000008000 - 0000009000] PGTABLE ==> [0000008000 - 0000009000] 2012-01-19 19:10:58 found SMP MP-table at [ffff8800000fe710] fe710 2012-01-19 19:10:58 Zone PFN ranges: 2012-01-19 19:10:58 DMA 0x00000001 -> 0x00001000 2012-01-19 19:10:58 DMA32 0x00001000 -> 0x00100000 2012-01-19 19:10:58 Normal 0x00100000 -> 0x00100000 2012-01-19 19:10:58 Movable zone start PFN for each node 2012-01-19 19:10:58 early_node_map[2] active PFN ranges 2012-01-19 19:10:58 1: 0x00000001 -> 0x000000a0 2012-01-19 19:10:58 1: 0x0000209f -> 0x0000dffb 2012-01-19 19:10:58 ACPI: PM-Timer IO Port: 0x808 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x01] lapic_id[0x10] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x02] lapic_id[0x00] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x03] lapic_id[0x12] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x04] lapic_id[0x02] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x05] lapic_id[0x14] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x06] lapic_id[0x04] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x07] lapic_id[0x16] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x08] lapic_id[0x06] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x09] lapic_id[0x11] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x01] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x13] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x03] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x15] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x05] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x17] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x10] lapic_id[0x07] enabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x11] lapic_id[0x30] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x12] lapic_id[0x31] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x13] lapic_id[0x32] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x14] lapic_id[0x33] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x15] lapic_id[0x34] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x16] lapic_id[0x35] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x17] lapic_id[0x36] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x18] lapic_id[0x37] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x19] lapic_id[0x38] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x39] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x3a] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x3b] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x3c] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x3d] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x3e] disabled) 2012-01-19 19:10:58 ACPI: LAPIC (acpi_id[0x20] lapic_id[0x3f] disabled) 2012-01-19 19:10:58 ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) 2012-01-19 19:10:58 ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) 2012-01-19 19:10:58 IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23 2012-01-19 19:10:58 ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32]) 2012-01-19 19:10:58 IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55 2012-01-19 19:10:58 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) 2012-01-19 19:10:58 ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) 2012-01-19 19:10:58 Using ACPI (MADT) for SMP configuration information 2012-01-19 19:10:58 ACPI: HPET id: 0x8086a301 base: 0xfed00000 2012-01-19 19:10:58 SMP: Allowing 32 CPUs, 16 hotplug CPUs 2012-01-19 19:10:58 PM: Registered nosave memory: 00000000000a0000 - 000000000209f000 2012-01-19 19:10:58 Allocating PCI resources starting at dffb000 (gap: dffb000:c169e000) 2012-01-19 19:10:58 Booting paravirtualized kernel on bare hardware 2012-01-19 19:10:58 NR_CPUS:4096 nr_cpumask_bits:32 nr_cpu_ids:32 nr_node_ids:2 2012-01-19 19:10:58 PERCPU: Embedded 30 pages/cpu @ffff880002a00000 s92568 r8192 d22120 u131072 2012-01-19 19:10:58 pcpu-alloc: s92568 r8192 d22120 u131072 alloc=1*2097152 2012-01-19 19:10:58 pcpu-alloc: [0] 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 2012-01-19 19:10:58 pcpu-alloc: [0] 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 2012-01-19 19:10:58 Built 1 zonelists in Node order, mobility grouping on. Total pages: 48262 2012-01-19 19:10:58 Policy zone: DMA32 2012-01-19 19:10:58 Kernel command line: nmi_watchdog=0 initrd=initramfs-2.6.32-131.17.1.el6_lustre.x86_64.img console=tty0 console=ttyS0,115200n8 BOOT_IMAGE=vmlinuz-2.6.32-131.17.1.el6_lustre.x86_64 BOOTIF=01-00-22-19-57-4b-03 irqpoll maxcpus=1 reset_devices cgroup_disable=memory memmap=exactmap memmap=636K@4K memmap=195952K@33404K elfcorehdr=229356K memmap=4K$0K memmap=88K$3398244K memmap=124K#3398332K memmap=9416K$3398456K memmap=262144K$3670016K memmap=32768K$4161536K 2012-01-19 19:10:58 Misrouted IRQ fixup and polling support enabled 2012-01-19 19:10:58 This may significantly impact system performance 2012-01-19 19:10:58 Disabling memory control group subsystem 2012-01-19 19:10:58 PID hash table entries: 1024 (order: 1, 8192 bytes) 2012-01-19 19:10:58 Checking aperture... 2012-01-19 19:10:58 No AGP bridge found 2012-01-19 19:10:58 AMD-Vi disabled by default: pass amd_iommu=on to enable 2012-01-19 19:10:58 Memory: 162020k/229356k available (5014k kernel code, 32768k absent, 34568k reserved, 6906k data, 1232k init) 2012-01-19 19:10:58 Hierarchical RCU implementation. 2012-01-19 19:10:58 NR_IRQS:33024 nr_irqs:1208 2012-01-19 19:10:58 Extended CMOS year: 2000 2012-01-19 19:10:58 Spurious LAPIC timer interrupt on cpu 0 2012-01-19 19:10:58 do_IRQ: 0.173 No irq handler for vector (irq -1) 2012-01-19 19:10:58 Console: colour VGA+ 80x25 2012-01-19 19:10:58 console [tty0] enabled 2012-01-19 19:10:58 console [ttyS0] enabled 2012-01-19 19:10:58 HPET: 4 timers in total, 0 timers will be used for per-cpu timer 2012-01-19 19:10:58 Fast TSC calibration using PIT 2012-01-19 19:10:58 Detected 2394.092 MHz processor. 2012-01-19 19:10:58 Calibrating delay loop (skipped), value calculated using timer frequency.. 4788.18 BogoMIPS (lpj=2394092) 2012-01-19 19:10:58 pid_max: default: 32768 minimum: 301 2012-01-19 19:10:58 Security Framework initialized 2012-01-19 19:10:58 SELinux: Initializing. 2012-01-19 19:10:58 Dentry cache hash table entries: 32768 (order: 6, 262144 bytes) 2012-01-19 19:10:58 Inode-cache hash table entries: 16384 (order: 5, 131072 bytes) 2012-01-19 19:10:58 Mount-cache hash table entries: 256 2012-01-19 19:10:58 Initializing cgroup subsys ns 2012-01-19 19:10:58 Initializing cgroup subsys cpuacct 2012-01-19 19:10:58 Initializing cgroup subsys memory 2012-01-19 19:10:58 Initializing cgroup subsys devices 2012-01-19 19:10:58 Initializing cgroup subsys freezer 2012-01-19 19:10:58 Initializing cgroup subsys net_cls 2012-01-19 19:10:58 Initializing cgroup subsys blkio 2012-01-19 19:10:58 CPU: Physical Processor ID: 0 2012-01-19 19:10:58 CPU: Processor Core ID: 2 2012-01-19 19:10:58 mce: CPU supports 9 MCE banks 2012-01-19 19:10:58 using mwait in idle threads. 2012-01-19 19:10:58 SMP alternatives: switching to UP code 2012-01-19 19:10:58 ACPI: Core revision 20090903 2012-01-19 19:10:58 ftrace: converting mcount calls to 0f 1f 44 00 00 2012-01-19 19:10:58 ftrace: allocating 20709 entries in 82 pages 2012-01-19 19:10:58 Setting APIC routing to physical flat 2012-01-19 19:10:58 ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 2012-01-19 19:10:58 CPU0: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05 2012-01-19 19:10:58 Performance Events: PEBS fmt1+, Nehalem events, Broken BIOS detected, complain to your hardware vendor. 2012-01-19 19:10:58 [Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR 186 is 53003c) 2012-01-19 19:10:58 Intel PMU driver. 2012-01-19 19:10:58 ... version: 3 2012-01-19 19:10:58 ... bit width: 48 2012-01-19 19:10:58 ... generic registers: 4 2012-01-19 19:10:58 ... value mask: 0000ffffffffffff 2012-01-19 19:10:58 ... max period: 000000007fffffff 2012-01-19 19:10:58 ... fixed-purpose events: 3 2012-01-19 19:10:58 ... event mask: 000000070000000f 2012-01-19 19:10:58 Brought up 1 CPUs 2012-01-19 19:10:58 Total of 1 processors activated (4788.18 BogoMIPS). 2012-01-19 19:10:58 devtmpfs: initialized 2012-01-19 19:10:58 regulator: core version 0.5 2012-01-19 19:10:58 NET: Registered protocol family 16 2012-01-19 19:10:58 ACPI FADT declares the system doesn't support PCIe ASPM, so disable it 2012-01-19 19:10:58 ACPI: bus type pci registered 2012-01-19 19:10:58 PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255 2012-01-19 19:10:58 PCI: MCFG area at e0000000 reserved in E820 2012-01-19 19:10:58 PCI: Using MMCONFIG at e0000000 - efffffff 2012-01-19 19:10:58 PCI: Using configuration type 1 for base access 2012-01-19 19:10:58 bio: create slab <bio-0> at 0 2012-01-19 19:10:58 ACPI Error (psargs-0359): [CDW1] Namespace lookup failure, AE_NOT_FOUND 2012-01-19 19:10:58 ACPI Error (psparse-0537): Method parse/execution failed [\_SB_._OSC] (Node ffff88000ccff768), AE_NOT_FOUND 2012-01-19 19:10:58 ACPI: BIOS _OSI(Linux) query ignored 2012-01-19 19:10:58 ACPI: Interpreter enabled 2012-01-19 19:10:58 ACPI: (supports S0 S4 S5) 2012-01-19 19:10:58 ACPI: Using IOAPIC for interrupt routing 2012-01-19 19:10:58 ACPI: No dock devices found. 2012-01-19 19:10:58 ACPI: PCI Root Bridge [PCI0] (0000:00) 2012-01-19 19:10:58 pci 0000:00:00.0: PME# supported from D0 D3hot D3cold 2012-01-19 19:10:58 pci 0000:00:00.0: PME# disabled 2012-01-19 19:10:58 pci 0000:00:01.0: PME# supported from D0 D3hot D3cold 2012-01-19 19:10:58 pci 0000:00:01.0: PME# disabled 2012-01-19 19:10:58 pci 0000:00:03.0: PME# supported from D0 D3hot D3cold 2012-01-19 19:10:58 pci 0000:00:03.0: PME# disabled 2012-01-19 19:10:58 pci 0000:00:07.0: PME# supported from D0 D3hot D3cold 2012-01-19 19:10:58 pci 0000:00:07.0: PME# disabled 2012-01-19 19:10:58 pci 0000:00:09.0: PME# supported from D0 D3hot D3cold 2012-01-19 19:10:58 pci 0000:00:09.0: PME# disabled 2012-01-19 19:10:58 pci 0000:00:1a.7: PME# supported from D0 D3hot D3cold 2012-01-19 19:10:58 pci 0000:00:1a.7: PME# disabled 2012-01-19 19:10:59 pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold 2012-01-19 19:10:59 pci 0000:00:1c.0: PME# disabled 2012-01-19 19:10:59 pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold 2012-01-19 19:10:59 pci 0000:00:1d.7: PME# disabled 2012-01-19 19:10:59 pci 0000:00:1f.0: quirk: region 0800-087f claimed by ICH6 ACPI/GPIO/TCO 2012-01-19 19:10:59 pci 0000:00:1f.0: quirk: region 0880-08bf claimed by ICH6 GPIO 2012-01-19 19:10:59 pci 0000:00:1f.0: ICH7 LPC Generic IO decode 1 PIO at 0c00 (mask 007f) 2012-01-19 19:10:59 pci 0000:00:1f.0: ICH7 LPC Generic IO decode 2 PIO at 0ca0 (mask 000f) 2012-01-19 19:10:59 pci 0000:00:1f.0: ICH7 LPC Generic IO decode 3 PIO at 00e0 (mask 000f) 2012-01-19 19:10:59 pci 0000:01:00.0: PME# supported from D0 D3hot D3cold 2012-01-19 19:10:59 pci 0000:01:00.0: PME# disabled 2012-01-19 19:10:59 pci 0000:01:00.1: PME# supported from D0 D3hot D3cold 2012-01-19 19:10:59 pci 0000:01:00.1: PME# disabled 2012-01-19 19:10:59 pci 0000:02:00.0: PME# supported from D0 D3hot D3cold 2012-01-19 19:10:59 pci 0000:02:00.0: PME# disabled 2012-01-19 19:10:59 pci 0000:02:00.1: PME# supported from D0 D3hot D3cold 2012-01-19 19:10:59 pci 0000:02:00.1: PME# disabled 2012-01-19 19:10:59 pci 0000:00:1e.0: transparent bridge 2012-01-19 19:10:59 ACPI: PCI Interrupt Link [LK00] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. 2012-01-19 19:10:59 ACPI: PCI Interrupt Link [LK01] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. 2012-01-19 19:10:59 ACPI: PCI Interrupt Link [LK02] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. 2012-01-19 19:10:59 ACPI: PCI Interrupt Link [LK03] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. 2012-01-19 19:10:59 ACPI: PCI Interrupt Link [LK04] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. 2012-01-19 19:10:59 ACPI: PCI Interrupt Link [LK05] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. 2012-01-19 19:10:59 ACPI: PCI Interrupt Link [LK06] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. 2012-01-19 19:10:59 ACPI: PCI Interrupt Link [LK07] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. 2012-01-19 19:10:59 HEST: HEST table parsing is initialized. 2012-01-19 19:10:59 vgaarb: device added: PCI:0000:06:03.0,decodes=io+mem,owns=io+mem,locks=none 2012-01-19 19:10:59 vgaarb: loaded 2012-01-19 19:10:59 SCSI subsystem initialized 2012-01-19 19:10:59 usbcore: registered new interface driver usbfs 2012-01-19 19:10:59 usbcore: registered new interface driver hub 2012-01-19 19:10:59 usbcore: registered new device driver usb 2012-01-19 19:10:59 PCI: Using ACPI for IRQ routing 2012-01-19 19:10:59 NetLabel: Initializing 2012-01-19 19:10:59 NetLabel: domain hash size = 128 2012-01-19 19:10:59 NetLabel: protocols = UNLABELED CIPSOv4 2012-01-19 19:10:59 NetLabel: unlabeled traffic allowed by default 2012-01-19 19:10:59 hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0 2012-01-19 19:10:59 hpet0: 4 comparators, 64-bit 14.318180 MHz counter 2012-01-19 19:10:59 Switching to clocksource tsc 2012-01-19 19:10:59 pnp: PnP ACPI init 2012-01-19 19:10:59 ACPI: bus type pnp registered 2012-01-19 19:10:59 pnp: PnP ACPI: found 13 devices 2012-01-19 19:10:59 ACPI: ACPI bus type pnp unregistered 2012-01-19 19:10:59 system 00:07: ioport range 0x800-0x87f has been reserved 2012-01-19 19:10:59 system 00:07: ioport range 0x880-0x8ff could not be reserved 2012-01-19 19:10:59 system 00:07: ioport range 0x900-0x91f has been reserved 2012-01-19 19:10:59 system 00:07: ioport range 0x920-0x923 has been reserved 2012-01-19 19:10:59 system 00:07: ioport range 0x924-0x924 has been reserved 2012-01-19 19:10:59 system 00:07: ioport range 0xc00-0xc7f has been reserved 2012-01-19 19:10:59 system 00:07: ioport range 0xca0-0xca7 has been reserved 2012-01-19 19:10:59 system 00:07: ioport range 0xca9-0xcab has been reserved 2012-01-19 19:10:59 system 00:07: ioport range 0xcad-0xcaf has been reserved 2012-01-19 19:10:59 system 00:08: ioport range 0xca8-0xca8 has been reserved 2012-01-19 19:10:59 system 00:08: ioport range 0xcac-0xcac has been reserved 2012-01-19 19:10:59 system 00:0a: iomem range 0xe0000000-0xefffffff has been reserved 2012-01-19 19:10:59 system 00:0c: iomem range 0xfed90000-0xfed91fff has been reserved 2012-01-19 19:10:59 pci 0000:00:01.0: PCI bridge to [bus 01-01] 2012-01-19 19:10:59 pci 0000:00:01.0: PCI bridge, secondary bus 0000:01 2012-01-19 19:10:59 pci 0000:00:01.0: bridge window [io disabled] 2012-01-19 19:10:59 pci 0000:00:01.0: bridge window [0xd6000000-0xd9ffffff] 2012-01-19 19:10:59 pci 0000:00:01.0: bridge window [mem pref disabled] 2012-01-19 19:10:59 pci 0000:00:03.0: PCI bridge to [bus 02-02] 2012-01-19 19:10:59 pci 0000:00:03.0: PCI bridge, secondary bus 0000:02 2012-01-19 19:10:59 pci 0000:00:03.0: bridge window [io disabled] 2012-01-19 19:10:59 pci 0000:00:03.0: bridge window [0xda000000-0xddffffff] 2012-01-19 19:10:59 pci 0000:00:03.0: bridge window [mem pref disabled] 2012-01-19 19:10:59 pci 0000:00:07.0: PCI bridge to [bus 04-04] 2012-01-19 19:10:59 pci 0000:00:07.0: PCI bridge, secondary bus 0000:04 2012-01-19 19:10:59 pci 0000:00:07.0: bridge window [io disabled] 2012-01-19 19:10:59 pci 0000:00:07.0: bridge window [0xdf100000-0xdf1fffff] 2012-01-19 19:10:59 pci 0000:00:07.0: bridge window [0xd4800000-0xd4ffffff] 2012-01-19 19:10:59 pci 0000:00:09.0: PCI bridge to [bus 05-05] 2012-01-19 19:10:59 pci 0000:00:09.0: PCI bridge, secondary bus 0000:05 2012-01-19 19:10:59 pci 0000:00:09.0: bridge window [io disabled] 2012-01-19 19:10:59 pci 0000:00:09.0: bridge window [0xdf200000-0xdf2fffff] 2012-01-19 19:10:59 pci 0000:00:09.0: bridge window [0xd5000000-0xd57fffff] 2012-01-19 19:10:59 pci 0000:00:1c.0: PCI bridge to [bus 03-03] 2012-01-19 19:10:59 pci 0000:00:1c.0: PCI bridge, secondary bus 0000:03 2012-01-19 19:10:59 pci 0000:00:1c.0: bridge window [0xf000-0xffff] 2012-01-19 19:10:59 pci 0000:00:1c.0: bridge window [0xdf300000-0xdf4fffff] 2012-01-19 19:10:59 pci 0000:00:1c.0: bridge window [mem pref disabled] 2012-01-19 19:10:59 pci 0000:00:1e.0: PCI bridge to [bus 06-06] 2012-01-19 19:10:59 pci 0000:00:1e.0: PCI bridge, secondary bus 0000:06 2012-01-19 19:10:59 pci 0000:00:1e.0: bridge window [io disabled] 2012-01-19 19:10:59 pci 0000:00:1e.0: bridge window [0xde000000-0xdeffffff] 2012-01-19 19:10:59 pci 0000:00:1e.0: bridge window [0xd5800000-0xd5ffffff] 2012-01-19 19:10:59 pci 0000:00:01.0: PCIe errors handled by BIOS. 2012-01-19 19:10:59 pci 0000:00:01.0: ACPI _OSC control granted for 0x14 2012-01-19 19:10:59 pci 0000:00:01.0: PCI INT A -> GSI 53 (level, low) -> IRQ 53 2012-01-19 19:10:59 pci 0000:00:03.0: PCIe errors handled by BIOS. 2012-01-19 19:10:59 pci 0000:00:03.0: ACPI _OSC control granted for 0x14 2012-01-19 19:10:59 pci 0000:00:03.0: PCI INT A -> GSI 53 (level, low) -> IRQ 53 2012-01-19 19:10:59 pci 0000:00:07.0: PCIe errors handled by BIOS. 2012-01-19 19:10:59 pci 0000:00:07.0: ACPI _OSC control granted for 0x14 2012-01-19 19:10:59 pci 0000:00:07.0: PCI INT A -> GSI 53 (level, low) -> IRQ 53 2012-01-19 19:10:59 pci 0000:00:09.0: PCIe errors handled by BIOS. 2012-01-19 19:10:59 pci 0000:00:09.0: ACPI _OSC control granted for 0x14 2012-01-19 19:10:59 pci 0000:00:09.0: PCI INT A -> GSI 53 (level, low) -> IRQ 53 2012-01-19 19:10:59 pci 0000:00:1c.0: PCIe errors handled by BIOS. 2012-01-19 19:10:59 pci 0000:00:1c.0: ACPI _OSC control granted for 0x14 2012-01-19 19:10:59 pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 2012-01-19 19:10:59 NET: Registered protocol family 2 2012-01-19 19:10:59 IP route cache hash table entries: 2048 (order: 2, 16384 bytes) 2012-01-19 19:10:59 TCP established hash table entries: 8192 (order: 5, 131072 bytes) 2012-01-19 19:10:59 TCP bind hash table entries: 8192 (order: 5, 131072 bytes) 2012-01-19 19:10:59 TCP: Hash tables configured (established 8192 bind 8192) 2012-01-19 19:10:59 TCP reno registered 2012-01-19 19:10:59 NET: Registered protocol family 1 2012-01-19 19:10:59 Trying to unpack rootfs image as initramfs... 2012-01-19 19:10:59 Freeing initrd memory: 10489k freed 2012-01-19 19:10:59 audit: initializing netlink socket (disabled) 2012-01-19 19:10:59 type=2000 audit(1327029057.743:1): initialized 2012-01-19 19:10:59 HugeTLB registered 2 MB page size, pre-allocated 0 pages 2012-01-19 19:10:59 VFS: Disk quotas dquot_6.5.2 2012-01-19 19:10:59 Dquot-cache hash table entries: 512 (order 0, 4096 bytes) 2012-01-19 19:10:59 msgmni has been set to 336 2012-01-19 19:10:59 alg: No test for stdrng (krng) 2012-01-19 19:10:59 ksign: Installing public key data 2012-01-19 19:10:59 Loading keyring 2012-01-19 19:10:59 - Added public key C60D77481C8E8402 2012-01-19 19:10:59 - User ID: Red Hat, Inc. (Kernel Module GPG key) 2012-01-19 19:10:59 - Added public key D4A26C9CCD09BEDA 2012-01-19 19:10:59 - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com> 2012-01-19 19:10:59 Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) 2012-01-19 19:10:59 io scheduler noop registered 2012-01-19 19:10:59 io scheduler anticipatory registered 2012-01-19 19:10:59 io scheduler deadline registered (default) 2012-01-19 19:10:59 io scheduler cfq registered 2012-01-19 19:10:59 pci_hotplug: PCI Hot Plug PCI Core version: 0.5 2012-01-19 19:10:59 pciehp: PCI Express Hot Plug Controller Driver version: 0.4 2012-01-19 19:10:59 acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 2012-01-19 19:10:59 pci-stub: invalid id string "" 2012-01-19 19:10:59 input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 2012-01-19 19:10:59 ACPI: Power Button [PWRF] 2012-01-19 19:10:59 processor LNXCPU:00: registered as cooling_device0 2012-01-19 19:10:59 processor LNXCPU:01: registered as cooling_device1 2012-01-19 19:10:59 processor LNXCPU:02: registered as cooling_device2 2012-01-19 19:10:59 processor LNXCPU:03: registered as cooling_device3 2012-01-19 19:10:59 processor LNXCPU:04: registered as cooling_device4 2012-01-19 19:10:59 processor LNXCPU:05: registered as cooling_device5 2012-01-19 19:10:59 processor LNXCPU:06: registered as cooling_device6 2012-01-19 19:10:59 processor LNXCPU:07: registered as cooling_device7 2012-01-19 19:10:59 processor LNXCPU:08: registered as cooling_device8 2012-01-19 19:10:59 processor LNXCPU:09: registered as cooling_device9 2012-01-19 19:10:59 processor LNXCPU:0a: registered as cooling_device10 2012-01-19 19:10:59 processor LNXCPU:0b: registered as cooling_device11 2012-01-19 19:10:59 processor LNXCPU:0c: registered as cooling_device12 2012-01-19 19:10:59 processor LNXCPU:0d: registered as cooling_device13 2012-01-19 19:10:59 processor LNXCPU:0e: registered as cooling_device14 2012-01-19 19:10:59 processor LNXCPU:0f: registered as cooling_device15 2012-01-19 19:10:59 ERST: Error Record Serialization Table (ERST) support is initialized. 2012-01-19 19:10:59 Non-volatile memory driver v1.3 2012-01-19 19:10:59 Linux agpgart interface v0.103 2012-01-19 19:10:59 crash memory driver: version 1.1 2012-01-19 19:10:59 Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled 2012-01-19 19:11:00 ~?serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A 2012-01-19 19:11:00 serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A 2012-01-19 19:11:00 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A 2012-01-19 19:11:00 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A 2012-01-19 19:11:00 brd: module loaded 2012-01-19 19:11:00 loop: module loaded 2012-01-19 19:11:00 input: Macintosh mouse button emulation as /devices/virtual/input/input1 2012-01-19 19:11:00 Fixed MDIO Bus: probed 2012-01-19 19:11:00 ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver 2012-01-19 19:11:00 ehci_hcd 0000:00:1a.7: PCI INT C -> GSI 19 (level, low) -> IRQ 19 2012-01-19 19:11:00 ehci_hcd 0000:00:1a.7: EHCI Host Controller 2012-01-19 19:11:00 ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 1 2012-01-19 19:11:00 ehci_hcd 0000:00:1a.7: debug port 1 2012-01-19 19:11:00 ehci_hcd 0000:00:1a.7: irq 19, io mem 0xdf0ff800 2012-01-19 19:11:00 ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00 2012-01-19 19:11:00 usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 2012-01-19 19:11:00 usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 2012-01-19 19:11:00 usb usb1: Product: EHCI Host Controller 2012-01-19 19:11:00 usb usb1: Manufacturer: Linux 2.6.32-131.17.1.el6_lustre.x86_64 ehci_hcd 2012-01-19 19:11:00 usb usb1: SerialNumber: 0000:00:1a.7 2012-01-19 19:11:00 usb usb1: configuration #1 chosen from 1 choice 2012-01-19 19:11:00 hub 1-0:1.0: USB hub found 2012-01-19 19:11:00 hub 1-0:1.0: 4 ports detected 2012-01-19 19:11:00 ehci_hcd 0000:00:1d.7: PCI INT A -> GSI 21 (level, low) -> IRQ 21 2012-01-19 19:11:00 ehci_hcd 0000:00:1d.7: EHCI Host Controller 2012-01-19 19:11:00 ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 2 2012-01-19 19:11:00 ehci_hcd 0000:00:1d.7: debug port 1 2012-01-19 19:11:00 ehci_hcd 0000:00:1d.7: irq 21, io mem 0xdf0ffc00 2012-01-19 19:11:00 ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00 2012-01-19 19:11:00 usb usb2: New USB device found, idVendor=1d6b, idProduct=0002 2012-01-19 19:11:00 usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 2012-01-19 19:11:00 usb usb2: Product: EHCI Host Controller 2012-01-19 19:11:00 usb usb2: Manufacturer: Linux 2.6.32-131.17.1.el6_lustre.x86_64 ehci_hcd 2012-01-19 19:11:00 usb usb2: SerialNumber: 0000:00:1d.7 2012-01-19 19:11:00 usb usb2: configuration #1 chosen from 1 choice 2012-01-19 19:11:00 hub 2-0:1.0: USB hub found 2012-01-19 19:11:00 hub 2-0:1.0: 4 ports detected 2012-01-19 19:11:00 ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver 2012-01-19 19:11:00 uhci_hcd: USB Universal Host Controller Interface driver 2012-01-19 19:11:00 uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17 2012-01-19 19:11:00 uhci_hcd 0000:00:1a.0: UHCI Host Controller 2012-01-19 19:11:00 uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3 2012-01-19 19:11:00 uhci_hcd 0000:00:1a.0: irq 17, io base 0x0000ec40 2012-01-19 19:11:00 usb usb3: New USB device found, idVendor=1d6b, idProduct=0001 2012-01-19 19:11:00 usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 2012-01-19 19:11:00 usb usb3: Product: UHCI Host Controller 2012-01-19 19:11:00 usb usb3: Manufacturer: Linux 2.6.32-131.17.1.el6_lustre.x86_64 uhci_hcd 2012-01-19 19:11:00 usb usb3: SerialNumber: 0000:00:1a.0 2012-01-19 19:11:00 usb usb3: configuration #1 chosen from 1 choice 2012-01-19 19:11:00 hub 3-0:1.0: USB hub found 2012-01-19 19:11:00 hub 3-0:1.0: 2 ports detected 2012-01-19 19:11:00 uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 18 (level, low) -> IRQ 18 2012-01-19 19:11:00 uhci_hcd 0000:00:1a.1: UHCI Host Controller 2012-01-19 19:11:00 uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4 2012-01-19 19:11:00 uhci_hcd 0000:00:1a.1: irq 18, io base 0x0000ec60 2012-01-19 19:11:00 usb usb4: New USB device found, idVendor=1d6b, idProduct=0001 2012-01-19 19:11:00 usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 2012-01-19 19:11:00 usb usb4: Product: UHCI Host Controller 2012-01-19 19:11:00 usb usb4: Manufacturer: Linux 2.6.32-131.17.1.el6_lustre.x86_64 uhci_hcd 2012-01-19 19:11:00 usb usb4: SerialNumber: 0000:00:1a.1 2012-01-19 19:11:00 usb usb4: configuration #1 chosen from 1 choice 2012-01-19 19:11:00 hub 4-0:1.0: USB hub found 2012-01-19 19:11:00 hub 4-0:1.0: 2 ports detected 2012-01-19 19:11:00 uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21 2012-01-19 19:11:00 uhci_hcd 0000:00:1d.0: UHCI Host Controller 2012-01-19 19:11:00 uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 5 2012-01-19 19:11:00 uhci_hcd 0000:00:1d.0: irq 21, io base 0x0000ec80 2012-01-19 19:11:00 usb usb5: New USB device found, idVendor=1d6b, idProduct=0001 2012-01-19 19:11:00 usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1 2012-01-19 19:11:00 usb usb5: Product: UHCI Host Controller 2012-01-19 19:11:00 usb usb5: Manufacturer: Linux 2.6.32-131.17.1.el6_lustre.x86_64 uhci_hcd 2012-01-19 19:11:00 usb usb5: SerialNumber: 0000:00:1d.0 2012-01-19 19:11:00 usb usb5: configuration #1 chosen from 1 choice 2012-01-19 19:11:00 hub 5-0:1.0: USB hub found 2012-01-19 19:11:00 hub 5-0:1.0: 2 ports detected 2012-01-19 19:11:00 uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 20 (level, low) -> IRQ 20 2012-01-19 19:11:00 uhci_hcd 0000:00:1d.1: UHCI Host Controller 2012-01-19 19:11:00 uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 6 2012-01-19 19:11:00 uhci_hcd 0000:00:1d.1: irq 20, io base 0x0000eca0 2012-01-19 19:11:00 usb usb6: New USB device found, idVendor=1d6b, idProduct=0001 2012-01-19 19:11:00 usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1 2012-01-19 19:11:00 usb usb6: Product: UHCI Host Controller 2012-01-19 19:11:00 usb usb6: Manufacturer: Linux 2.6.32-131.17.1.el6_lustre.x86_64 uhci_hcd 2012-01-19 19:11:00 usb usb6: SerialNumber: 0000:00:1d.1 2012-01-19 19:11:00 usb usb6: configuration #1 chosen from 1 choice 2012-01-19 19:11:00 hub 6-0:1.0: USB hub found 2012-01-19 19:11:00 hub 6-0:1.0: 2 ports detected 2012-01-19 19:11:00 PNP: No PS/2 controller found. Probing ports directly. 2012-01-19 19:11:00 usb 1-3: new high speed USB device using ehci_hcd and address 2 2012-01-19 19:11:00 serio: i8042 KBD port at 0x60,0x64 irq 1 2012-01-19 19:11:00 serio: i8042 AUX port at 0x60,0x64 irq 12 2012-01-19 19:11:00 mice: PS/2 mouse device common for all mice 2012-01-19 19:11:00 rtc_cmos 00:04: RTC can wake from S4 2012-01-19 19:11:00 rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0 2012-01-19 19:11:00 rtc0: alarms up to one day, y3k, 242 bytes nvram, hpet irqs 2012-01-19 19:11:00 cpuidle: using governor ladder 2012-01-19 19:11:00 cpuidle: using governor menu 2012-01-19 19:11:00 usbcore: registered new interface driver hiddev 2012-01-19 19:11:00 usbcore: registered new interface driver usbhid 2012-01-19 19:11:00 usbhid: v2.6:USB HID core driver 2012-01-19 19:11:00 TCP cubic registered 2012-01-19 19:11:00 Initializing XFRM netlink socket 2012-01-19 19:11:00 NET: Registered protocol family 17 2012-01-19 19:11:00 registered taskstats version 1 2012-01-19 19:11:00 rtc_cmos 00:04: setting system clock to 2012-01-20 03:11:00 UTC (1327029060) 2012-01-19 19:11:00 Initalizing network drop monitor service 2012-01-19 19:11:00 Freeing unused kernel memory: 1232k freed 2012-01-19 19:11:00 Write protecting the kernel read-only data: 10240k 2012-01-19 19:11:00 Freeing unused kernel memory: 1112k freed 2012-01-19 19:11:00 Freeing unused kernel memory: 1796k freed 2012-01-19 19:11:00 udev: starting version 147 2012-01-19 19:11:00 usb 1-3: New USB device found, idVendor=0424, idProduct=2514 2012-01-19 19:11:00 usb 1-3: New USB device strings: Mfr=0, Product=0, SerialNumber=0 2012-01-19 19:11:01 bnx2: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.1.6 (Mar 7, 2011) 2012-01-19 19:11:01 usb 1-3: configuration #1 chosen from 1 choice 2012-01-19 19:11:01 bnx2 0000:01:00.0: PCI INT A -> GSI 36 (level, low) -> IRQ 36 2012-01-19 19:11:01 hub 1-3:1.0: USB hub found 2012-01-19 19:11:01 bnx2 0000:01:00.0: firmware: requesting bnx2/bnx2-mips-09-6.2.1a.fw 2012-01-19 19:11:01 hub 1-3:1.0: 3 ports detected 2012-01-19 19:11:01 bnx2 0000:01:00.0: firmware: requesting bnx2/bnx2-rv2p-09-6.0.17.fw 2012-01-19 19:11:01 bnx2 0000:01:00.0: eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem d6000000, IRQ 36, node addr 00:22:19:57:4b:03 2012-01-19 19:11:01 bnx2 0000:01:00.1: PCI INT B -> GSI 48 (level, low) -> IRQ 48 2012-01-19 19:11:01 bnx2 0000:01:00.1: firmware: requesting bnx2/bnx2-mips-09-6.2.1a.fw 2012-01-19 19:11:01 bnx2 0000:01:00.1: firmware: requesting bnx2/bnx2-rv2p-09-6.0.17.fw 2012-01-19 19:11:01 bnx2 0000:01:00.1: eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem d8000000, IRQ 48, node addr 00:22:19:57:4b:05 2012-01-19 19:11:01 bnx2 0000:02:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32 2012-01-19 19:11:01 bnx2 0000:02:00.0: firmware: requesting bnx2/bnx2-mips-09-6.2.1a.fw 2012-01-19 19:11:01 bnx2 0000:02:00.0: firmware: requesting bnx2/bnx2-rv2p-09-6.0.17.fw 2012-01-19 19:11:01 bnx2 0000:02:00.0: eth2: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem da000000, IRQ 32, node addr 00:22:19:57:4b:07 2012-01-19 19:11:01 bnx2 0000:02:00.1: PCI INT B -> GSI 42 (level, low) -> IRQ 42 2012-01-19 19:11:01 bnx2 0000:02:00.1: firmware: requesting bnx2/bnx2-mips-09-6.2.1a.fw 2012-01-19 19:11:01 bnx2 0000:02:00.1: firmware: requesting bnx2/bnx2-rv2p-09-6.0.17.fw 2012-01-19 19:11:01 bnx2 0000:02:00.1: eth3: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem dc000000, IRQ 42, node addr 00:22:19:57:4b:09 2012-01-19 19:11:01 mlx4_core: Mellanox ConnectX core driver v0.01 (May 1, 2007) 2012-01-19 19:11:01 mlx4_core: Initializing 0000:04:00.0 2012-01-19 19:11:01 mlx4_core 0000:04:00.0: PCI INT A -> GSI 38 (level, low) -> IRQ 38 2012-01-19 19:11:01 bnx2 0000:01:00.0: eth0: using MSIX 2012-01-19 19:11:01 usb 5-2: new full speed USB device using uhci_hcd and address 2 2012-01-19 19:11:01 usb 5-2: New USB device found, idVendor=0624, idProduct=0248 2012-01-19 19:11:01 usb 5-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 2012-01-19 19:11:01 usb 5-2: Product: USB Composite Device-0 2012-01-19 19:11:01 usb 5-2: Manufacturer: Avocent 2012-01-19 19:11:01 usb 5-2: SerialNumber: 20080519 2012-01-19 19:11:01 usb 5-2: configuration #1 chosen from 1 choice 2012-01-19 19:11:01 input: Avocent USB Composite Device-0 as /devices/pci0000:00/0000:00:1d.0/usb5/5-2/5-2:1.0/input/input2 2012-01-19 19:11:01 generic-usb 0003:0624:0248.0001: input,hidraw0: USB HID v1.00 Keyboard [Avocent USB Composite Device-0] on usb-0000:00:1d.0-2/input0 2012-01-19 19:11:01 input: Avocent USB Composite Device-0 as /devices/pci0000:00/0000:00:1d.0/usb5/5-2/5-2:1.1/input/input3 2012-01-19 19:11:01 generic-usb 0003:0624:0248.0002: input,hidraw1: USB HID v1.00 Mouse [Avocent USB Composite Device-0] on usb-0000:00:1d.0-2/input1 2012-01-19 19:11:03 modprobe invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=-17 2012-01-19 19:11:03 modprobe cpuset=/ mems_allowed=1 2012-01-19 19:11:03 Pid: 252, comm: modprobe Not tainted 2.6.32-131.17.1.el6_lustre.x86_64 #1 2012-01-19 19:11:03 Call Trace: 2012-01-19 19:11:03 [<ffffffff810c00f1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 2012-01-19 19:11:03 [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0 2012-01-19 19:11:03 [<ffffffff81110880>] ? select_bad_process+0xd0/0x110 2012-01-19 19:11:03 [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0 2012-01-19 19:11:03 [<ffffffff81110b19>] ? out_of_memory+0x199/0x210 2012-01-19 19:11:03 [<ffffffff81120262>] ? __alloc_pages_nodemask+0x812/0x8b0 2012-01-19 19:11:03 [<ffffffff81010e86>] ? dma_generic_alloc_coherent+0xa6/0x160 2012-01-19 19:11:03 [<ffffffffa0017e99>] ? mlx4_create_eq+0x139/0x6b0 [mlx4_core] 2012-01-19 19:11:03 [<ffffffffa001857d>] ? mlx4_init_eq_table+0x16d/0x560 [mlx4_core] 2012-01-19 19:11:03 [<ffffffffa00185f9>] ? mlx4_init_eq_table+0x1e9/0x560 [mlx4_core] 2012-01-19 19:11:03 [<ffffffffa001d4d0>] ? mlx4_setup_hca+0xa0/0x5c0 [mlx4_core] 2012-01-19 19:11:03 [<ffffffffa001e045>] ? __mlx4_init_one+0x2f5/0x880 [mlx4_core] 2012-01-19 19:11:03 [<ffffffffa002315f>] ? mlx4_init_one+0x42/0x47 [mlx4_core] 2012-01-19 19:11:03 [<ffffffff81281107>] ? local_pci_probe+0x17/0x20 2012-01-19 19:11:03 [<ffffffff812822f1>] ? pci_device_probe+0x101/0x120 2012-01-19 19:11:03 [<ffffffff8133bb92>] ? driver_sysfs_add+0x62/0x90 2012-01-19 19:11:03 [<ffffffff8133bd30>] ? driver_probe_device+0xa0/0x2a0 2012-01-19 19:11:03 [<ffffffff8133bfdb>] ? __driver_attach+0xab/0xb0 2012-01-19 19:11:03 [<ffffffff8133bf30>] ? __driver_attach+0x0/0xb0 2012-01-19 19:11:03 [<ffffffff8133af94>] ? bus_for_each_dev+0x64/0x90 2012-01-19 19:11:03 [<ffffffff8133bace>] ? driver_attach+0x1e/0x20 2012-01-19 19:11:03 [<ffffffff8133b3d0>] ? bus_add_driver+0x200/0x300 2012-01-19 19:11:03 [<ffffffff8133c306>] ? driver_register+0x76/0x140 2012-01-19 19:11:03 [<ffffffff810899a8>] ? __create_workqueue_key+0x1e8/0x280 2012-01-19 19:11:03 [<ffffffff81282556>] ? __pci_register_driver+0x56/0xd0 2012-01-19 19:11:03 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:03 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:03 [<ffffffffa002e0b2>] ? mlx4_init+0x81/0xbf [mlx4_core] 2012-01-19 19:11:03 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 2012-01-19 19:11:03 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 2012-01-19 19:11:03 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b 2012-01-19 19:11:03 Mem-Info: 2012-01-19 19:11:03 Node 1 DMA per-cpu: 2012-01-19 19:11:03 CPU 0: hi: 0, btch: 1 usd: 0 2012-01-19 19:11:03 Node 1 DMA32 per-cpu: 2012-01-19 19:11:03 CPU 0: hi: 90, btch: 15 usd: 100 2012-01-19 19:11:03 active_anon:2987 inactive_anon:177 isolated_anon:0 2012-01-19 19:11:03 active_file:3 inactive_file:0 isolated_file:0 2012-01-19 19:11:03 unevictable:6725 dirty:0 writeback:0 unstable:0 2012-01-19 19:11:03 free:532 slab_reclaimable:1202 slab_unreclaimable:4819 2012-01-19 19:11:03 mapped:492 shmem:23 pagetables:892 bounce:0 2012-01-19 19:11:03 Node 1 DMA free:388kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2012-01-19 19:11:03 lowmem_reserve[]: 0 188 188 188 2012-01-19 19:11:03 Node 1 DMA32 free:1740kB min:1752kB low:2188kB high:2628kB active_anon:11948kB inactive_anon:708kB active_file:12kB inactive_file:0kB unevictable:26900kB isolated(anon):0kB isolated(file):0kB present:193040kB mlocked:0kB dirty:0kB writeback:0kB mapped:1968kB shmem:92kB slab_reclaimable:4808kB slab_unreclaimable:19276kB kernel_stack:1168kB pagetables:3568kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-01-19 19:11:03 lowmem_reserve[]: 0 0 0 0 2012-01-19 19:11:03 Node 1 DMA: 1*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 388kB 2012-01-19 19:11:03 Node 1 DMA32: 51*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1740kB 2012-01-19 19:11:03 6749 total pagecache pages 2012-01-19 19:11:03 0 pages in swap cache 2012-01-19 19:11:03 Swap cache stats: add 0, delete 0, find 0/0 2012-01-19 19:11:03 Free swap = 0kB 2012-01-19 19:11:03 Total swap = 0kB 2012-01-19 19:11:03 57338 pages RAM 2012-01-19 19:11:03 13175 pages reserved 2012-01-19 19:11:03 18205 pages shared 2012-01-19 19:11:03 35137 pages non-shared 2012-01-19 19:11:03 Out of memory: kill process 259 (dhclient) score 573 or a child 2012-01-19 19:11:03 Killed process 259 (dhclient) vsz:9168kB, anon-rss:464kB, file-rss:912kB 2012-01-19 19:11:03 modprobe invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=-17 2012-01-19 19:11:03 modprobe cpuset=/ mems_allowed=1 2012-01-19 19:11:03 Pid: 252, comm: modprobe Not tainted 2.6.32-131.17.1.el6_lustre.x86_64 #1 2012-01-19 19:11:03 Call Trace: 2012-01-19 19:11:03 [<ffffffff810c00f1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 2012-01-19 19:11:03 [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0 2012-01-19 19:11:03 [<ffffffff81110880>] ? select_bad_process+0xd0/0x110 2012-01-19 19:11:03 [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0 2012-01-19 19:11:03 [<ffffffff81110b19>] ? out_of_memory+0x199/0x210 2012-01-19 19:11:03 [<ffffffff81120262>] ? __alloc_pages_nodemask+0x812/0x8b0 2012-01-19 19:11:03 [<ffffffff81010e86>] ? dma_generic_alloc_coherent+0xa6/0x160 2012-01-19 19:11:03 [<ffffffffa0017e99>] ? mlx4_create_eq+0x139/0x6b0 [mlx4_core] 2012-01-19 19:11:03 [<ffffffffa001857d>] ? mlx4_init_eq_table+0x16d/0x560 [mlx4_core] 2012-01-19 19:11:03 [<ffffffffa00185f9>] ? mlx4_init_eq_table+0x1e9/0x560 [mlx4_core] 2012-01-19 19:11:03 [<ffffffffa001d4d0>] ? mlx4_setup_hca+0xa0/0x5c0 [mlx4_core] 2012-01-19 19:11:03 [<ffffffffa001e045>] ? __mlx4_init_one+0x2f5/0x880 [mlx4_core] 2012-01-19 19:11:03 [<ffffffffa002315f>] ? mlx4_init_one+0x42/0x47 [mlx4_core] 2012-01-19 19:11:03 [<ffffffff81281107>] ? local_pci_probe+0x17/0x20 2012-01-19 19:11:03 [<ffffffff812822f1>] ? pci_device_probe+0x101/0x120 2012-01-19 19:11:03 [<ffffffff8133bb92>] ? driver_sysfs_add+0x62/0x90 2012-01-19 19:11:03 [<ffffffff8133bd30>] ? driver_probe_device+0xa0/0x2a0 2012-01-19 19:11:03 [<ffffffff8133bfdb>] ? __driver_attach+0xab/0xb0 2012-01-19 19:11:03 [<ffffffff8133bf30>] ? __driver_attach+0x0/0xb0 2012-01-19 19:11:03 [<ffffffff8133af94>] ? bus_for_each_dev+0x64/0x90 2012-01-19 19:11:03 [<ffffffff8133bace>] ? driver_attach+0x1e/0x20 2012-01-19 19:11:03 [<ffffffff8133b3d0>] ? bus_add_driver+0x200/0x300 2012-01-19 19:11:03 [<ffffffff8133c306>] ? driver_register+0x76/0x140 2012-01-19 19:11:03 [<ffffffff810899a8>] ? __create_workqueue_key+0x1e8/0x280 2012-01-19 19:11:03 [<ffffffff81282556>] ? __pci_register_driver+0x56/0xd0 2012-01-19 19:11:03 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:03 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:03 [<ffffffffa002e0b2>] ? mlx4_init+0x81/0xbf [mlx4_core] 2012-01-19 19:11:03 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 2012-01-19 19:11:03 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 2012-01-19 19:11:03 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b 2012-01-19 19:11:03 Mem-Info: 2012-01-19 19:11:03 Node 1 DMA per-cpu: 2012-01-19 19:11:03 CPU 0: hi: 0, btch: 1 usd: 0 2012-01-19 19:11:03 Node 1 DMA32 per-cpu: 2012-01-19 19:11:03 CPU 0: hi: 90, btch: 15 usd: 83 2012-01-19 19:11:04 active_anon:1496 inactive_anon:1552 isolated_anon:0 2012-01-19 19:11:04 active_file:0 inactive_file:0 isolated_file:0 2012-01-19 19:11:04 unevictable:6726 dirty:0 writeback:0 unstable:0 2012-01-19 19:11:04 free:530 slab_reclaimable:1189 slab_unreclaimable:4816 2012-01-19 19:11:04 mapped:411 shmem:23 pagetables:886 bounce:0 2012-01-19 19:11:04 Node 1 DMA free:388kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2012-01-19 19:11:04 lowmem_reserve[]: 0 188 188 188 2012-01-19 19:11:04 Node 1 DMA32 free:1732kB min:1752kB low:2188kB high:2628kB active_anon:5984kB inactive_anon:6208kB active_file:0kB inactive_file:0kB unevictable:26904kB isolated(anon):0kB isolated(file):0kB present:193040kB mlocked:0kB dirty:0kB writeback:0kB mapped:1644kB shmem:92kB slab_reclaimable:4756kB slab_unreclaimable:19264kB kernel_stack:1152kB pagetables:3544kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1344 all_unreclaimable? yes 2012-01-19 19:11:04 lowmem_reserve[]: 0 0 0 0 2012-01-19 19:11:04 Node 1 DMA: 1*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 388kB 2012-01-19 19:11:04 Node 1 DMA32: 53*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1732kB 2012-01-19 19:11:04 6749 total pagecache pages 2012-01-19 19:11:04 0 pages in swap cache 2012-01-19 19:11:04 Swap cache stats: add 0, delete 0, find 0/0 2012-01-19 19:11:04 Free swap = 0kB 2012-01-19 19:11:04 Total swap = 0kB 2012-01-19 19:11:04 57338 pages RAM 2012-01-19 19:11:04 13175 pages reserved 2012-01-19 19:11:04 17977 pages shared 2012-01-19 19:11:04 35237 pages non-shared 2012-01-19 19:11:04 Out of memory: kill process 266 (udevadm) score 132 or a child 2012-01-19 19:11:04 Killed process 266 (udevadm) vsz:8492kB, anon-rss:120kB, file-rss:664kB 2012-01-19 19:11:04 /lib/dracut-lib.sh: line 157: 266 Killed udevadm settle --eximodprobe invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=-17 2012-01-19 19:11:04 t-if-exists=/inimodprobe cpuset=/ mems_allowed=1 2012-01-19 19:11:04 tqueue/work $setPid: 252, comm: modprobe Not tainted 2.6.32-131.17.1.el6_lustre.x86_64 #1 2012-01-19 19:11:04 tle_exit_if_exisCall Trace: 2012-01-19 19:11:04 ts 2012-01-19 19:11:04 [<ffffffff810c00f1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 2012-01-19 19:11:04 [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0 2012-01-19 19:11:04 [<ffffffff81110880>] ? select_bad_process+0xd0/0x110 2012-01-19 19:11:04 [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0 2012-01-19 19:11:04 [<ffffffff81110b19>] ? out_of_memory+0x199/0x210 2012-01-19 19:11:04 [<ffffffff81120262>] ? __alloc_pages_nodemask+0x812/0x8b0 2012-01-19 19:11:04 [<ffffffff81010e86>] ? dma_generic_alloc_coherent+0xa6/0x160 2012-01-19 19:11:04 [<ffffffffa0017e99>] ? mlx4_create_eq+0x139/0x6b0 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa001857d>] ? mlx4_init_eq_table+0x16d/0x560 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa00185f9>] ? mlx4_init_eq_table+0x1e9/0x560 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa001d4d0>] ? mlx4_setup_hca+0xa0/0x5c0 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa001e045>] ? __mlx4_init_one+0x2f5/0x880 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa002315f>] ? mlx4_init_one+0x42/0x47 [mlx4_core] 2012-01-19 19:11:04 [<ffffffff81281107>] ? local_pci_probe+0x17/0x20 2012-01-19 19:11:04 [<ffffffff812822f1>] ? pci_device_probe+0x101/0x120 2012-01-19 19:11:04 [<ffffffff8133bb92>] ? driver_sysfs_add+0x62/0x90 2012-01-19 19:11:04 [<ffffffff8133bd30>] ? driver_probe_device+0xa0/0x2a0 2012-01-19 19:11:04 [<ffffffff8133bfdb>] ? __driver_attach+0xab/0xb0 2012-01-19 19:11:04 [<ffffffff8133bf30>] ? __driver_attach+0x0/0xb0 2012-01-19 19:11:04 [<ffffffff8133af94>] ? bus_for_each_dev+0x64/0x90 2012-01-19 19:11:04 [<ffffffff8133bace>] ? driver_attach+0x1e/0x20 2012-01-19 19:11:04 [<ffffffff8133b3d0>] ? bus_add_driver+0x200/0x300 2012-01-19 19:11:04 [<ffffffff8133c306>] ? driver_register+0x76/0x140 2012-01-19 19:11:04 [<ffffffff810899a8>] ? __create_workqueue_key+0x1e8/0x280 2012-01-19 19:11:04 [<ffffffff81282556>] ? __pci_register_driver+0x56/0xd0 2012-01-19 19:11:04 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa002e0b2>] ? mlx4_init+0x81/0xbf [mlx4_core] 2012-01-19 19:11:04 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 2012-01-19 19:11:04 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 2012-01-19 19:11:04 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b 2012-01-19 19:11:04 Mem-Info: 2012-01-19 19:11:04 Node 1 DMA per-cpu: 2012-01-19 19:11:04 CPU 0: hi: 0, btch: 1 usd: 0 2012-01-19 19:11:04 Node 1 DMA32 per-cpu: 2012-01-19 19:11:04 CPU 0: hi: 90, btch: 15 usd: 15 2012-01-19 19:11:04 active_anon:1501 inactive_anon:1547 isolated_anon:0 2012-01-19 19:11:04 active_file:0 inactive_file:0 isolated_file:0 2012-01-19 19:11:04 unevictable:6726 dirty:0 writeback:0 unstable:0 2012-01-19 19:11:04 free:528 slab_reclaimable:1189 slab_unreclaimable:4816 2012-01-19 19:11:04 mapped:411 shmem:23 pagetables:886 bounce:0 2012-01-19 19:11:04 Node 1 DMA free:388kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2012-01-19 19:11:04 lowmem_reserve[]: 0 188 188 188 2012-01-19 19:11:04 Node 1 DMA32 free:1724kB min:1752kB low:2188kB high:2628kB active_anon:6004kB inactive_anon:6188kB active_file:0kB inactive_file:0kB unevictable:26904kB isolated(anon):0kB isolated(file):0kB present:193040kB mlocked:0kB dirty:0kB writeback:0kB mapped:1644kB shmem:92kB slab_reclaimable:4756kB slab_unreclaimable:19264kB kernel_stack:1152kB pagetables:3544kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-01-19 19:11:04 lowmem_reserve[]: 0 0 0 0 2012-01-19 19:11:04 Node 1 DMA: 1*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 388kB 2012-01-19 19:11:04 Node 1 DMA32: 31*4kB 3*8kB 3*16kB 2*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1732kB 2012-01-19 19:11:04 6749 total pagecache pages 2012-01-19 19:11:04 0 pages in swap cache 2012-01-19 19:11:04 Swap cache stats: add 0, delete 0, find 0/0 2012-01-19 19:11:04 Free swap = 0kB 2012-01-19 19:11:04 Total swap = 0kB 2012-01-19 19:11:04 57338 pages RAM 2012-01-19 19:11:04 13175 pages reserved 2012-01-19 19:11:04 17977 pages shared 2012-01-19 19:11:04 35304 pages non-shared 2012-01-19 19:11:04 Out of memory: kill process 313 (udevadm) score 132 or a child 2012-01-19 19:11:04 Killed process 313 (udevadm) vsz:8492kB, anon-rss:120kB, file-rss:660kB 2012-01-19 19:11:04 /lib/dracut-lib.modprobe invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=-17 2012-01-19 19:11:04 sh: line 157: modprobe cpuset=/ mems_allowed=1 2012-01-19 19:11:04 313 Killed Pid: 252, comm: modprobe Not tainted 2.6.32-131.17.1.el6_lustre.x86_64 #1 2012-01-19 19:11:04 udevCall Trace: 2012-01-19 19:11:04 adm settle --exi [<ffffffff810c00f1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 2012-01-19 19:11:04 t-if-exists=/ini [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0 2012-01-19 19:11:04 tqueue/work $set [<ffffffff81110880>] ? select_bad_process+0xd0/0x110 2012-01-19 19:11:04 tle_exit_if_exis [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0 2012-01-19 19:11:04 ts 2012-01-19 19:11:04 [<ffffffff81110b19>] ? out_of_memory+0x199/0x210 2012-01-19 19:11:04 [<ffffffff81120262>] ? __alloc_pages_nodemask+0x812/0x8b0 2012-01-19 19:11:04 [<ffffffff81010e86>] ? dma_generic_alloc_coherent+0xa6/0x160 2012-01-19 19:11:04 [<ffffffffa0017e99>] ? mlx4_create_eq+0x139/0x6b0 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa001857d>] ? mlx4_init_eq_table+0x16d/0x560 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa00185f9>] ? mlx4_init_eq_table+0x1e9/0x560 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa001d4d0>] ? mlx4_setup_hca+0xa0/0x5c0 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa001e045>] ? __mlx4_init_one+0x2f5/0x880 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa002315f>] ? mlx4_init_one+0x42/0x47 [mlx4_core] 2012-01-19 19:11:04 [<ffffffff81281107>] ? local_pci_probe+0x17/0x20 2012-01-19 19:11:04 [<ffffffff812822f1>] ? pci_device_probe+0x101/0x120 2012-01-19 19:11:04 [<ffffffff8133bb92>] ? driver_sysfs_add+0x62/0x90 2012-01-19 19:11:04 [<ffffffff8133bd30>] ? driver_probe_device+0xa0/0x2a0 2012-01-19 19:11:04 [<ffffffff8133bfdb>] ? __driver_attach+0xab/0xb0 2012-01-19 19:11:04 [<ffffffff8133bf30>] ? __driver_attach+0x0/0xb0 2012-01-19 19:11:04 [<ffffffff8133af94>] ? bus_for_each_dev+0x64/0x90 2012-01-19 19:11:04 [<ffffffff8133bace>] ? driver_attach+0x1e/0x20 2012-01-19 19:11:04 [<ffffffff8133b3d0>] ? bus_add_driver+0x200/0x300 2012-01-19 19:11:04 [<ffffffff8133c306>] ? driver_register+0x76/0x140 2012-01-19 19:11:04 [<ffffffff810899a8>] ? __create_workqueue_key+0x1e8/0x280 2012-01-19 19:11:04 [<ffffffff81282556>] ? __pci_register_driver+0x56/0xd0 2012-01-19 19:11:04 bnx2 0000:01:00.0: eth0: NIC Copper Link is Up, 1000 Mbps full duplex 2012-01-19 19:11:04 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa002e0b2>] ? mlx4_init+0x81/0xbf [mlx4_core] 2012-01-19 19:11:04 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 2012-01-19 19:11:04 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 2012-01-19 19:11:04 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b 2012-01-19 19:11:04 Mem-Info: 2012-01-19 19:11:04 Node 1 DMA per-cpu: 2012-01-19 19:11:04 CPU 0: hi: 0, btch: 1 usd: 0 2012-01-19 19:11:04 Node 1 DMA32 per-cpu: 2012-01-19 19:11:04 CPU 0: hi: 90, btch: 15 usd: 28 2012-01-19 19:11:04 active_anon:1486 inactive_anon:1547 isolated_anon:0 2012-01-19 19:11:04 active_file:0 inactive_file:0 isolated_file:0 2012-01-19 19:11:04 unevictable:6726 dirty:0 writeback:0 unstable:0 2012-01-19 19:11:04 free:526 slab_reclaimable:1189 slab_unreclaimable:4816 2012-01-19 19:11:04 mapped:390 shmem:23 pagetables:886 bounce:0 2012-01-19 19:11:04 Node 1 DMA free:388kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2012-01-19 19:11:04 lowmem_reserve[]: 0 188 188 188 2012-01-19 19:11:04 Node 1 DMA32 free:1716kB min:1752kB low:2188kB high:2628kB active_anon:5944kB inactive_anon:6188kB active_file:0kB inactive_file:0kB unevictable:26904kB isolated(anon):0kB isolated(file):0kB present:193040kB mlocked:0kB dirty:0kB writeback:0kB mapped:1560kB shmem:92kB slab_reclaimable:4756kB slab_unreclaimable:19264kB kernel_stack:1152kB pagetables:3544kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-01-19 19:11:04 lowmem_reserve[]: 0 0 0 0 2012-01-19 19:11:04 Node 1 DMA: 1*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 388kB 2012-01-19 19:11:04 Node 1 DMA32: 29*4kB 7*8kB 3*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1724kB 2012-01-19 19:11:04 6749 total pagecache pages 2012-01-19 19:11:04 0 pages in swap cache 2012-01-19 19:11:04 Swap cache stats: add 0, delete 0, find 0/0 2012-01-19 19:11:04 Free swap = 0kB 2012-01-19 19:11:04 Total swap = 0kB 2012-01-19 19:11:04 57338 pages RAM 2012-01-19 19:11:04 13175 pages reserved 2012-01-19 19:11:04 17926 pages shared 2012-01-19 19:11:04 35218 pages non-shared 2012-01-19 19:11:04 Out of memory: kill process 314 (init) score 181 or a child 2012-01-19 19:11:04 Killed process 314 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:04 /lib/dracut-lib.modprobe invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=-17 2012-01-19 19:11:04 sh: line 73: 3modprobe cpuset=/ mems_allowed=1 2012-01-19 19:11:04 14 Killed Pid: 252, comm: modprobe Not tainted 2.6.32-131.17.1.el6_lustre.x86_64 #1 2012-01-19 19:11:04 ( . "Call Trace: 2012-01-19 19:11:04 $f" ) 2012-01-19 19:11:04 [<ffffffff810c00f1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 2012-01-19 19:11:04 [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0 2012-01-19 19:11:04 [<ffffffff81110880>] ? select_bad_process+0xd0/0x110 2012-01-19 19:11:04 [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0 2012-01-19 19:11:04 [<ffffffff81110b19>] ? out_of_memory+0x199/0x210 2012-01-19 19:11:04 [<ffffffff81120262>] ? __alloc_pages_nodemask+0x812/0x8b0 2012-01-19 19:11:04 [<ffffffff81010e86>] ? dma_generic_alloc_coherent+0xa6/0x160 2012-01-19 19:11:04 [<ffffffffa0017e99>] ? mlx4_create_eq+0x139/0x6b0 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa001857d>] ? mlx4_init_eq_table+0x16d/0x560 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa00185f9>] ? mlx4_init_eq_table+0x1e9/0x560 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa001d4d0>] ? mlx4_setup_hca+0xa0/0x5c0 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa001e045>] ? __mlx4_init_one+0x2f5/0x880 [mlx4_core] 2012-01-19 19:11:04 [<ffffffffa002315f>] ? mlx4_init_one+0x42/0x47 [mlx4_core] 2012-01-19 19:11:04 [<ffffffff81281107>] ? local_pci_probe+0x17/0x20 2012-01-19 19:11:04 [<ffffffff812822f1>] ? pci_device_probe+0x101/0x120 2012-01-19 19:11:04 [<ffffffff8133bb92>] ? driver_sysfs_add+0x62/0x90 2012-01-19 19:11:04 [<ffffffff8133bd30>] ? driver_probe_device+0xa0/0x2a0 2012-01-19 19:11:04 [<ffffffff8133bfdb>] ? __driver_attach+0xab/0xb0 2012-01-19 19:11:04 [<ffffffff8133bf30>] ? __driver_attach+0x0/0xb0 2012-01-19 19:11:04 [<ffffffff8133af94>] ? bus_for_each_dev+0x64/0x90 2012-01-19 19:11:04 [<ffffffff8133bace>] ? driver_attach+0x1e/0x20 2012-01-19 19:11:04 [<ffffffff8133b3d0>] ? bus_add_driver+0x200/0x300 2012-01-19 19:11:05 [<ffffffff8133c306>] ? driver_register+0x76/0x140 2012-01-19 19:11:05 [<ffffffff810899a8>] ? __create_workqueue_key+0x1e8/0x280 2012-01-19 19:11:05 [<ffffffff81282556>] ? __pci_register_driver+0x56/0xd0 2012-01-19 19:11:05 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa002e0b2>] ? mlx4_init+0x81/0xbf [mlx4_core] 2012-01-19 19:11:05 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 2012-01-19 19:11:05 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 2012-01-19 19:11:05 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b 2012-01-19 19:11:05 Mem-Info: 2012-01-19 19:11:05 Node 1 DMA per-cpu: 2012-01-19 19:11:05 CPU 0: hi: 0, btch: 1 usd: 0 2012-01-19 19:11:05 Node 1 DMA32 per-cpu: 2012-01-19 19:11:05 CPU 0: hi: 90, btch: 15 usd: 28 2012-01-19 19:11:05 active_anon:1498 inactive_anon:1546 isolated_anon:0 2012-01-19 19:11:05 active_file:0 inactive_file:0 isolated_file:0 2012-01-19 19:11:05 unevictable:6725 dirty:0 writeback:0 unstable:0 2012-01-19 19:11:05 free:522 slab_reclaimable:1174 slab_unreclaimable:4820 2012-01-19 19:11:05 mapped:386 shmem:23 pagetables:882 bounce:0 2012-01-19 19:11:05 Node 1 DMA free:388kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2012-01-19 19:11:05 lowmem_reserve[]: 0 188 188 188 2012-01-19 19:11:05 Node 1 DMA32 free:1700kB min:1752kB low:2188kB high:2628kB active_anon:5992kB inactive_anon:6184kB active_file:0kB inactive_file:0kB unevictable:26900kB isolated(anon):0kB isolated(file):0kB present:193040kB mlocked:0kB dirty:0kB writeback:0kB mapped:1544kB shmem:92kB slab_reclaimable:4696kB slab_unreclaimable:19280kB kernel_stack:1144kB pagetables:3528kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-01-19 19:11:05 lowmem_reserve[]: 0 0 0 0 2012-01-19 19:11:05 Node 1 DMA: 1*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 388kB 2012-01-19 19:11:05 Node 1 DMA32: 29*4kB 5*8kB 3*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1708kB 2012-01-19 19:11:05 6748 total pagecache pages 2012-01-19 19:11:05 0 pages in swap cache 2012-01-19 19:11:05 Swap cache stats: add 0, delete 0, find 0/0 2012-01-19 19:11:05 Free swap = 0kB 2012-01-19 19:11:05 Total swap = 0kB 2012-01-19 19:11:05 57338 pages RAM 2012-01-19 19:11:05 13175 pages reserved 2012-01-19 19:11:05 17934 pages shared 2012-01-19 19:11:05 35236 pages non-shared 2012-01-19 19:11:05 Out of memory: kill process 315 (init) score 181 or a child 2012-01-19 19:11:05 Killed process 315 (init) vsz:11624kB, anon-rss:464kB, file-rss:132kB 2012-01-19 19:11:05 /init: line 221:modprobe invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=-17 2012-01-19 19:11:05 315 Killed modprobe cpuset=/ mems_allowed=1 2012-01-19 19:11:05 $Pid: 252, comm: modprobe Not tainted 2.6.32-131.17.1.el6_lustre.x86_64 #1 2012-01-19 19:11:05 UDEV_QUEUE_EMPTYCall Trace: 2012-01-19 19:11:05 > /dev/null 2>& [<ffffffff810c00f1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 2012-01-19 19:11:05 1 2012-01-19 19:11:05 [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0 2012-01-19 19:11:05 [<ffffffff81110880>] ? select_bad_process+0xd0/0x110 2012-01-19 19:11:05 [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0 2012-01-19 19:11:05 [<ffffffff81110b19>] ? out_of_memory+0x199/0x210 2012-01-19 19:11:05 [<ffffffff81120262>] ? __alloc_pages_nodemask+0x812/0x8b0 2012-01-19 19:11:05 [<ffffffff81010e86>] ? dma_generic_alloc_coherent+0xa6/0x160 2012-01-19 19:11:05 [<ffffffffa0017e99>] ? mlx4_create_eq+0x139/0x6b0 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa001857d>] ? mlx4_init_eq_table+0x16d/0x560 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa00185f9>] ? mlx4_init_eq_table+0x1e9/0x560 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa001d4d0>] ? mlx4_setup_hca+0xa0/0x5c0 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa001e045>] ? __mlx4_init_one+0x2f5/0x880 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa002315f>] ? mlx4_init_one+0x42/0x47 [mlx4_core] 2012-01-19 19:11:05 [<ffffffff81281107>] ? local_pci_probe+0x17/0x20 2012-01-19 19:11:05 [<ffffffff812822f1>] ? pci_device_probe+0x101/0x120 2012-01-19 19:11:05 [<ffffffff8133bb92>] ? driver_sysfs_add+0x62/0x90 2012-01-19 19:11:05 [<ffffffff8133bd30>] ? driver_probe_device+0xa0/0x2a0 2012-01-19 19:11:05 [<ffffffff8133bfdb>] ? __driver_attach+0xab/0xb0 2012-01-19 19:11:05 [<ffffffff8133bf30>] ? __driver_attach+0x0/0xb0 2012-01-19 19:11:05 [<ffffffff8133af94>] ? bus_for_each_dev+0x64/0x90 2012-01-19 19:11:05 [<ffffffff8133bace>] ? driver_attach+0x1e/0x20 2012-01-19 19:11:05 [<ffffffff8133b3d0>] ? bus_add_driver+0x200/0x300 2012-01-19 19:11:05 [<ffffffff8133c306>] ? driver_register+0x76/0x140 2012-01-19 19:11:05 [<ffffffff810899a8>] ? __create_workqueue_key+0x1e8/0x280 2012-01-19 19:11:05 [<ffffffff81282556>] ? __pci_register_driver+0x56/0xd0 2012-01-19 19:11:05 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa002e0b2>] ? mlx4_init+0x81/0xbf [mlx4_core] 2012-01-19 19:11:05 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 2012-01-19 19:11:05 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 2012-01-19 19:11:05 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b 2012-01-19 19:11:05 Mem-Info: 2012-01-19 19:11:05 Node 1 DMA per-cpu: 2012-01-19 19:11:05 CPU 0: hi: 0, btch: 1 usd: 0 2012-01-19 19:11:05 Node 1 DMA32 per-cpu: 2012-01-19 19:11:05 CPU 0: hi: 90, btch: 15 usd: 28 2012-01-19 19:11:05 active_anon:1498 inactive_anon:1546 isolated_anon:0 2012-01-19 19:11:05 active_file:0 inactive_file:0 isolated_file:0 2012-01-19 19:11:05 unevictable:6725 dirty:0 writeback:0 unstable:0 2012-01-19 19:11:05 free:522 slab_reclaimable:1174 slab_unreclaimable:4820 2012-01-19 19:11:05 mapped:386 shmem:23 pagetables:882 bounce:0 2012-01-19 19:11:05 Node 1 DMA free:388kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2012-01-19 19:11:05 lowmem_reserve[]: 0 188 188 188 2012-01-19 19:11:05 Node 1 DMA32 free:1700kB min:1752kB low:2188kB high:2628kB active_anon:5992kB inactive_anon:6184kB active_file:0kB inactive_file:0kB unevictable:26900kB isolated(anon):0kB isolated(file):0kB present:193040kB mlocked:0kB dirty:0kB writeback:0kB mapped:1544kB shmem:92kB slab_reclaimable:4696kB slab_unreclaimable:19280kB kernel_stack:1144kB pagetables:3528kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-01-19 19:11:05 lowmem_reserve[]: 0 0 0 0 2012-01-19 19:11:05 Node 1 DMA: 1*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 388kB 2012-01-19 19:11:05 Node 1 DMA32: 29*4kB 5*8kB 3*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1708kB 2012-01-19 19:11:05 6748 total pagecache pages 2012-01-19 19:11:05 0 pages in swap cache 2012-01-19 19:11:05 Swap cache stats: add 0, delete 0, find 0/0 2012-01-19 19:11:05 Free swap = 0kB 2012-01-19 19:11:05 Total swap = 0kB 2012-01-19 19:11:05 57338 pages RAM 2012-01-19 19:11:05 13175 pages reserved 2012-01-19 19:11:05 17922 pages shared 2012-01-19 19:11:05 35236 pages non-shared 2012-01-19 19:11:05 Out of memory: kill process 316 (init) score 181 or a child 2012-01-19 19:11:05 Killed process 316 (init) vsz:11624kB, anon-rss:464kB, file-rss:84kB 2012-01-19 19:11:05 /lib/dracut-lib.modprobe invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=-17 2012-01-19 19:11:05 sh: line 73: 3modprobe cpuset=/ mems_allowed=1 2012-01-19 19:11:05 16 Killed Pid: 252, comm: modprobe Not tainted 2.6.32-131.17.1.el6_lustre.x86_64 #1 2012-01-19 19:11:05 ( . "Call Trace: 2012-01-19 19:11:05 $f" ) 2012-01-19 19:11:05 [<ffffffff810c00f1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 2012-01-19 19:11:05 [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0 2012-01-19 19:11:05 [<ffffffff81110880>] ? select_bad_process+0xd0/0x110 2012-01-19 19:11:05 [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0 2012-01-19 19:11:05 [<ffffffff81110b19>] ? out_of_memory+0x199/0x210 2012-01-19 19:11:05 [<ffffffff81120262>] ? __alloc_pages_nodemask+0x812/0x8b0 2012-01-19 19:11:05 [<ffffffff81010e86>] ? dma_generic_alloc_coherent+0xa6/0x160 2012-01-19 19:11:05 [<ffffffffa0017e99>] ? mlx4_create_eq+0x139/0x6b0 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa001857d>] ? mlx4_init_eq_table+0x16d/0x560 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa00185f9>] ? mlx4_init_eq_table+0x1e9/0x560 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa001d4d0>] ? mlx4_setup_hca+0xa0/0x5c0 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa001e045>] ? __mlx4_init_one+0x2f5/0x880 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa002315f>] ? mlx4_init_one+0x42/0x47 [mlx4_core] 2012-01-19 19:11:05 [<ffffffff81281107>] ? local_pci_probe+0x17/0x20 2012-01-19 19:11:05 [<ffffffff812822f1>] ? pci_device_probe+0x101/0x120 2012-01-19 19:11:05 [<ffffffff8133bb92>] ? driver_sysfs_add+0x62/0x90 2012-01-19 19:11:05 [<ffffffff8133bd30>] ? driver_probe_device+0xa0/0x2a0 2012-01-19 19:11:05 [<ffffffff8133bfdb>] ? __driver_attach+0xab/0xb0 2012-01-19 19:11:05 [<ffffffff8133bf30>] ? __driver_attach+0x0/0xb0 2012-01-19 19:11:05 [<ffffffff8133af94>] ? bus_for_each_dev+0x64/0x90 2012-01-19 19:11:05 [<ffffffff8133bace>] ? driver_attach+0x1e/0x20 2012-01-19 19:11:05 [<ffffffff8133b3d0>] ? bus_add_driver+0x200/0x300 2012-01-19 19:11:05 [<ffffffff8133c306>] ? driver_register+0x76/0x140 2012-01-19 19:11:05 [<ffffffff810899a8>] ? __create_workqueue_key+0x1e8/0x280 2012-01-19 19:11:05 [<ffffffff81282556>] ? __pci_register_driver+0x56/0xd0 2012-01-19 19:11:05 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa002e0b2>] ? mlx4_init+0x81/0xbf [mlx4_core] 2012-01-19 19:11:05 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 2012-01-19 19:11:05 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 2012-01-19 19:11:05 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b 2012-01-19 19:11:05 Mem-Info: 2012-01-19 19:11:05 Node 1 DMA per-cpu: 2012-01-19 19:11:05 CPU 0: hi: 0, btch: 1 usd: 0 2012-01-19 19:11:05 Node 1 DMA32 per-cpu: 2012-01-19 19:11:05 CPU 0: hi: 90, btch: 15 usd: 33 2012-01-19 19:11:05 active_anon:1503 inactive_anon:1546 isolated_anon:0 2012-01-19 19:11:05 active_file:0 inactive_file:0 isolated_file:0 2012-01-19 19:11:05 unevictable:6725 dirty:0 writeback:0 unstable:0 2012-01-19 19:11:05 free:522 slab_reclaimable:1174 slab_unreclaimable:4820 2012-01-19 19:11:05 mapped:400 shmem:23 pagetables:882 bounce:0 2012-01-19 19:11:05 Node 1 DMA free:388kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2012-01-19 19:11:05 lowmem_reserve[]: 0 188 188 188 2012-01-19 19:11:05 Node 1 DMA32 free:1700kB min:1752kB low:2188kB high:2628kB active_anon:6012kB inactive_anon:6184kB active_file:0kB inactive_file:0kB unevictable:26900kB isolated(anon):0kB isolated(file):0kB present:193040kB mlocked:0kB dirty:0kB writeback:0kB mapped:1600kB shmem:92kB slab_reclaimable:4696kB slab_unreclaimable:19280kB kernel_stack:1144kB pagetables:3528kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-01-19 19:11:05 lowmem_reserve[]: 0 0 0 0 2012-01-19 19:11:05 Node 1 DMA: 1*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 388kB 2012-01-19 19:11:05 Node 1 DMA32: 29*4kB 5*8kB 3*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1708kB 2012-01-19 19:11:05 6748 total pagecache pages 2012-01-19 19:11:05 0 pages in swap cache 2012-01-19 19:11:05 Swap cache stats: add 0, delete 0, find 0/0 2012-01-19 19:11:05 Free swap = 0kB 2012-01-19 19:11:05 Total swap = 0kB 2012-01-19 19:11:05 57338 pages RAM 2012-01-19 19:11:05 13175 pages reserved 2012-01-19 19:11:05 17951 pages shared 2012-01-19 19:11:05 35301 pages non-shared 2012-01-19 19:11:05 Out of memory: kill process 317 (udevadm) score 132 or a child 2012-01-19 19:11:05 Killed process 317 (udevadm) vsz:8492kB, anon-rss:116kB, file-rss:556kB 2012-01-19 19:11:05 /lib/dracut-lib.modprobe invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=-17 2012-01-19 19:11:05 sh: line 157: modprobe cpuset=/ mems_allowed=1 2012-01-19 19:11:05 317 Killed Pid: 252, comm: modprobe Not tainted 2.6.32-131.17.1.el6_lustre.x86_64 #1 2012-01-19 19:11:05 udevCall Trace: 2012-01-19 19:11:05 adm settle --exi [<ffffffff810c00f1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 2012-01-19 19:11:05 t-if-exists=/ini [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0 2012-01-19 19:11:05 tqueue/work $set [<ffffffff81110880>] ? select_bad_process+0xd0/0x110 2012-01-19 19:11:05 tle_exit_if_exis [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0 2012-01-19 19:11:05 ts 2012-01-19 19:11:05 [<ffffffff81110b19>] ? out_of_memory+0x199/0x210 2012-01-19 19:11:05 [<ffffffff81120262>] ? __alloc_pages_nodemask+0x812/0x8b0 2012-01-19 19:11:05 [<ffffffff81010e86>] ? dma_generic_alloc_coherent+0xa6/0x160 2012-01-19 19:11:05 [<ffffffffa0017e99>] ? mlx4_create_eq+0x139/0x6b0 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa001857d>] ? mlx4_init_eq_table+0x16d/0x560 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa00185f9>] ? mlx4_init_eq_table+0x1e9/0x560 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa001d4d0>] ? mlx4_setup_hca+0xa0/0x5c0 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa001e045>] ? __mlx4_init_one+0x2f5/0x880 [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa002315f>] ? mlx4_init_one+0x42/0x47 [mlx4_core] 2012-01-19 19:11:05 [<ffffffff81281107>] ? local_pci_probe+0x17/0x20 2012-01-19 19:11:05 [<ffffffff812822f1>] ? pci_device_probe+0x101/0x120 2012-01-19 19:11:05 [<ffffffff8133bb92>] ? driver_sysfs_add+0x62/0x90 2012-01-19 19:11:05 [<ffffffff8133bd30>] ? driver_probe_device+0xa0/0x2a0 2012-01-19 19:11:05 [<ffffffff8133bfdb>] ? __driver_attach+0xab/0xb0 2012-01-19 19:11:05 [<ffffffff8133bf30>] ? __driver_attach+0x0/0xb0 2012-01-19 19:11:05 [<ffffffff8133af94>] ? bus_for_each_dev+0x64/0x90 2012-01-19 19:11:05 [<ffffffff8133bace>] ? driver_attach+0x1e/0x20 2012-01-19 19:11:05 [<ffffffff8133b3d0>] ? bus_add_driver+0x200/0x300 2012-01-19 19:11:05 [<ffffffff8133c306>] ? driver_register+0x76/0x140 2012-01-19 19:11:05 [<ffffffff810899a8>] ? __create_workqueue_key+0x1e8/0x280 2012-01-19 19:11:05 [<ffffffff81282556>] ? __pci_register_driver+0x56/0xd0 2012-01-19 19:11:05 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:05 [<ffffffffa002e0b2>] ? mlx4_init+0x81/0xbf [mlx4_core] 2012-01-19 19:11:05 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 2012-01-19 19:11:05 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 2012-01-19 19:11:05 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b 2012-01-19 19:11:05 Mem-Info: 2012-01-19 19:11:05 Node 1 DMA per-cpu: 2012-01-19 19:11:05 CPU 0: hi: 0, btch: 1 usd: 0 2012-01-19 19:11:05 Node 1 DMA32 per-cpu: 2012-01-19 19:11:06 CPU 0: hi: 90, btch: 15 usd: 28 2012-01-19 19:11:06 active_anon:1498 inactive_anon:1546 isolated_anon:0 2012-01-19 19:11:06 active_file:0 inactive_file:0 isolated_file:0 2012-01-19 19:11:06 unevictable:6725 dirty:0 writeback:0 unstable:0 2012-01-19 19:11:06 free:522 slab_reclaimable:1174 slab_unreclaimable:4820 2012-01-19 19:11:06 mapped:386 shmem:23 pagetables:882 bounce:0 2012-01-19 19:11:06 Node 1 DMA free:388kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2012-01-19 19:11:06 lowmem_reserve[]: 0 188 188 188 2012-01-19 19:11:06 Node 1 DMA32 free:1700kB min:1752kB low:2188kB high:2628kB active_anon:5992kB inactive_anon:6184kB active_file:0kB inactive_file:0kB unevictable:26900kB isolated(anon):0kB isolated(file):0kB present:193040kB mlocked:0kB dirty:0kB writeback:0kB mapped:1544kB shmem:92kB slab_reclaimable:4696kB slab_unreclaimable:19280kB kernel_stack:1144kB pagetables:3528kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-01-19 19:11:06 lowmem_reserve[]: 0 0 0 0 2012-01-19 19:11:06 Node 1 DMA: 1*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 388kB 2012-01-19 19:11:06 Node 1 DMA32: 27*4kB 6*8kB 3*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1708kB 2012-01-19 19:11:06 6748 total pagecache pages 2012-01-19 19:11:06 0 pages in swap cache 2012-01-19 19:11:06 Swap cache stats: add 0, delete 0, find 0/0 2012-01-19 19:11:06 Free swap = 0kB 2012-01-19 19:11:06 Total swap = 0kB 2012-01-19 19:11:06 57338 pages RAM 2012-01-19 19:11:06 13175 pages reserved 2012-01-19 19:11:06 17935 pages shared 2012-01-19 19:11:06 35236 pages non-shared 2012-01-19 19:11:06 Out of memory: kill process 318 (init) score 181 or a child 2012-01-19 19:11:06 Killed process 318 (init) vsz:11624kB, anon-rss:464kB, file-rss:136kB 2012-01-19 19:11:06 /lib/dracut-lib.modprobe invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=-17 2012-01-19 19:11:06 sh: line 73: 3modprobe cpuset=/ mems_allowed=1 2012-01-19 19:11:06 18 Killed Pid: 252, comm: modprobe Not tainted 2.6.32-131.17.1.el6_lustre.x86_64 #1 2012-01-19 19:11:06 ( . "Call Trace: 2012-01-19 19:11:06 $f" ) 2012-01-19 19:11:06 [<ffffffff810c00f1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 2012-01-19 19:11:06 [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0 2012-01-19 19:11:06 [<ffffffff81110880>] ? select_bad_process+0xd0/0x110 2012-01-19 19:11:06 [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0 2012-01-19 19:11:06 [<ffffffff81110b19>] ? out_of_memory+0x199/0x210 2012-01-19 19:11:06 [<ffffffff81120262>] ? __alloc_pages_nodemask+0x812/0x8b0 2012-01-19 19:11:06 [<ffffffff81010e86>] ? dma_generic_alloc_coherent+0xa6/0x160 2012-01-19 19:11:06 [<ffffffffa0017e99>] ? mlx4_create_eq+0x139/0x6b0 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa001857d>] ? mlx4_init_eq_table+0x16d/0x560 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa00185f9>] ? mlx4_init_eq_table+0x1e9/0x560 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa001d4d0>] ? mlx4_setup_hca+0xa0/0x5c0 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa001e045>] ? __mlx4_init_one+0x2f5/0x880 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa002315f>] ? mlx4_init_one+0x42/0x47 [mlx4_core] 2012-01-19 19:11:06 [<ffffffff81281107>] ? local_pci_probe+0x17/0x20 2012-01-19 19:11:06 [<ffffffff812822f1>] ? pci_device_probe+0x101/0x120 2012-01-19 19:11:06 [<ffffffff8133bb92>] ? driver_sysfs_add+0x62/0x90 2012-01-19 19:11:06 [<ffffffff8133bd30>] ? driver_probe_device+0xa0/0x2a0 2012-01-19 19:11:06 [<ffffffff8133bfdb>] ? __driver_attach+0xab/0xb0 2012-01-19 19:11:06 [<ffffffff8133bf30>] ? __driver_attach+0x0/0xb0 2012-01-19 19:11:06 [<ffffffff8133af94>] ? bus_for_each_dev+0x64/0x90 2012-01-19 19:11:06 [<ffffffff8133bace>] ? driver_attach+0x1e/0x20 2012-01-19 19:11:06 [<ffffffff8133b3d0>] ? bus_add_driver+0x200/0x300 2012-01-19 19:11:06 [<ffffffff8133c306>] ? driver_register+0x76/0x140 2012-01-19 19:11:06 [<ffffffff810899a8>] ? __create_workqueue_key+0x1e8/0x280 2012-01-19 19:11:06 [<ffffffff81282556>] ? __pci_register_driver+0x56/0xd0 2012-01-19 19:11:06 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa002e0b2>] ? mlx4_init+0x81/0xbf [mlx4_core] 2012-01-19 19:11:06 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 2012-01-19 19:11:06 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 2012-01-19 19:11:06 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b 2012-01-19 19:11:06 Mem-Info: 2012-01-19 19:11:06 Node 1 DMA per-cpu: 2012-01-19 19:11:06 CPU 0: hi: 0, btch: 1 usd: 0 2012-01-19 19:11:06 Node 1 DMA32 per-cpu: 2012-01-19 19:11:06 CPU 0: hi: 90, btch: 15 usd: 28 2012-01-19 19:11:06 active_anon:1498 inactive_anon:1546 isolated_anon:0 2012-01-19 19:11:06 active_file:0 inactive_file:0 isolated_file:0 2012-01-19 19:11:06 unevictable:6725 dirty:0 writeback:0 unstable:0 2012-01-19 19:11:06 free:522 slab_reclaimable:1174 slab_unreclaimable:4820 2012-01-19 19:11:06 mapped:386 shmem:23 pagetables:882 bounce:0 2012-01-19 19:11:06 Node 1 DMA free:388kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2012-01-19 19:11:06 lowmem_reserve[]: 0 188 188 188 2012-01-19 19:11:06 Node 1 DMA32 free:1700kB min:1752kB low:2188kB high:2628kB active_anon:5992kB inactive_anon:6184kB active_file:0kB inactive_file:0kB unevictable:26900kB isolated(anon):0kB isolated(file):0kB present:193040kB mlocked:0kB dirty:0kB writeback:0kB mapped:1544kB shmem:92kB slab_reclaimable:4696kB slab_unreclaimable:19280kB kernel_stack:1144kB pagetables:3528kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-01-19 19:11:06 lowmem_reserve[]: 0 0 0 0 2012-01-19 19:11:06 Node 1 DMA: 1*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 388kB 2012-01-19 19:11:06 Node 1 DMA32: 27*4kB 6*8kB 3*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1708kB 2012-01-19 19:11:06 6748 total pagecache pages 2012-01-19 19:11:06 0 pages in swap cache 2012-01-19 19:11:06 Swap cache stats: add 0, delete 0, find 0/0 2012-01-19 19:11:06 Free swap = 0kB 2012-01-19 19:11:06 Total swap = 0kB 2012-01-19 19:11:06 57338 pages RAM 2012-01-19 19:11:06 13175 pages reserved 2012-01-19 19:11:06 17928 pages shared 2012-01-19 19:11:06 35236 pages non-shared 2012-01-19 19:11:06 Out of memory: kill process 319 (init) score 181 or a child 2012-01-19 19:11:06 Killed process 319 (init) vsz:11624kB, anon-rss:464kB, file-rss:108kB 2012-01-19 19:11:06 /init: line 221:modprobe invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=-17 2012-01-19 19:11:06 319 Killed modprobe cpuset=/ mems_allowed=1 2012-01-19 19:11:06 $Pid: 252, comm: modprobe Not tainted 2.6.32-131.17.1.el6_lustre.x86_64 #1 2012-01-19 19:11:06 UDEV_QUEUE_EMPTYCall Trace: 2012-01-19 19:11:06 > /dev/null 2>& [<ffffffff810c00f1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 2012-01-19 19:11:06 1 2012-01-19 19:11:06 [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0 2012-01-19 19:11:06 [<ffffffff81110880>] ? select_bad_process+0xd0/0x110 2012-01-19 19:11:06 [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0 2012-01-19 19:11:06 [<ffffffff81110b19>] ? out_of_memory+0x199/0x210 2012-01-19 19:11:06 [<ffffffff81120262>] ? __alloc_pages_nodemask+0x812/0x8b0 2012-01-19 19:11:06 [<ffffffff81010e86>] ? dma_generic_alloc_coherent+0xa6/0x160 2012-01-19 19:11:06 [<ffffffffa0017e99>] ? mlx4_create_eq+0x139/0x6b0 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa001857d>] ? mlx4_init_eq_table+0x16d/0x560 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa00185f9>] ? mlx4_init_eq_table+0x1e9/0x560 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa001d4d0>] ? mlx4_setup_hca+0xa0/0x5c0 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa001e045>] ? __mlx4_init_one+0x2f5/0x880 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa002315f>] ? mlx4_init_one+0x42/0x47 [mlx4_core] 2012-01-19 19:11:06 [<ffffffff81281107>] ? local_pci_probe+0x17/0x20 2012-01-19 19:11:06 [<ffffffff812822f1>] ? pci_device_probe+0x101/0x120 2012-01-19 19:11:06 [<ffffffff8133bb92>] ? driver_sysfs_add+0x62/0x90 2012-01-19 19:11:06 [<ffffffff8133bd30>] ? driver_probe_device+0xa0/0x2a0 2012-01-19 19:11:06 [<ffffffff8133bfdb>] ? __driver_attach+0xab/0xb0 2012-01-19 19:11:06 [<ffffffff8133bf30>] ? __driver_attach+0x0/0xb0 2012-01-19 19:11:06 [<ffffffff8133af94>] ? bus_for_each_dev+0x64/0x90 2012-01-19 19:11:06 [<ffffffff8133bace>] ? driver_attach+0x1e/0x20 2012-01-19 19:11:06 [<ffffffff8133b3d0>] ? bus_add_driver+0x200/0x300 2012-01-19 19:11:06 [<ffffffff8133c306>] ? driver_register+0x76/0x140 2012-01-19 19:11:06 [<ffffffff810899a8>] ? __create_workqueue_key+0x1e8/0x280 2012-01-19 19:11:06 [<ffffffff81282556>] ? __pci_register_driver+0x56/0xd0 2012-01-19 19:11:06 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa002e0b2>] ? mlx4_init+0x81/0xbf [mlx4_core] 2012-01-19 19:11:06 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 2012-01-19 19:11:06 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 2012-01-19 19:11:06 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b 2012-01-19 19:11:06 Mem-Info: 2012-01-19 19:11:06 Node 1 DMA per-cpu: 2012-01-19 19:11:06 CPU 0: hi: 0, btch: 1 usd: 0 2012-01-19 19:11:06 Node 1 DMA32 per-cpu: 2012-01-19 19:11:06 CPU 0: hi: 90, btch: 15 usd: 28 2012-01-19 19:11:06 active_anon:1498 inactive_anon:1546 isolated_anon:0 2012-01-19 19:11:06 active_file:0 inactive_file:0 isolated_file:0 2012-01-19 19:11:06 unevictable:6725 dirty:0 writeback:0 unstable:0 2012-01-19 19:11:06 free:522 slab_reclaimable:1174 slab_unreclaimable:4820 2012-01-19 19:11:06 mapped:386 shmem:23 pagetables:882 bounce:0 2012-01-19 19:11:06 Node 1 DMA free:388kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2012-01-19 19:11:06 lowmem_reserve[]: 0 188 188 188 2012-01-19 19:11:06 Node 1 DMA32 free:1700kB min:1752kB low:2188kB high:2628kB active_anon:5992kB inactive_anon:6184kB active_file:0kB inactive_file:0kB unevictable:26900kB isolated(anon):0kB isolated(file):0kB present:193040kB mlocked:0kB dirty:0kB writeback:0kB mapped:1544kB shmem:92kB slab_reclaimable:4696kB slab_unreclaimable:19280kB kernel_stack:1144kB pagetables:3528kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-01-19 19:11:06 lowmem_reserve[]: 0 0 0 0 2012-01-19 19:11:06 Node 1 DMA: 1*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 388kB 2012-01-19 19:11:06 Node 1 DMA32: 27*4kB 6*8kB 3*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1708kB 2012-01-19 19:11:06 6748 total pagecache pages 2012-01-19 19:11:06 0 pages in swap cache 2012-01-19 19:11:06 Swap cache stats: add 0, delete 0, find 0/0 2012-01-19 19:11:06 Free swap = 0kB 2012-01-19 19:11:06 Total swap = 0kB 2012-01-19 19:11:06 57338 pages RAM 2012-01-19 19:11:06 13175 pages reserved 2012-01-19 19:11:06 17922 pages shared 2012-01-19 19:11:06 35236 pages non-shared 2012-01-19 19:11:06 Out of memory: kill process 320 (init) score 181 or a child 2012-01-19 19:11:06 Killed process 320 (init) vsz:11624kB, anon-rss:464kB, file-rss:84kB 2012-01-19 19:11:06 /lib/dracut-lib.modprobe invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=-17 2012-01-19 19:11:06 sh: line 73: 3modprobe cpuset=/ mems_allowed=1 2012-01-19 19:11:06 20 Killed Pid: 252, comm: modprobe Not tainted 2.6.32-131.17.1.el6_lustre.x86_64 #1 2012-01-19 19:11:06 ( . "Call Trace: 2012-01-19 19:11:06 $f" ) 2012-01-19 19:11:06 [<ffffffff810c00f1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 2012-01-19 19:11:06 [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0 2012-01-19 19:11:06 [<ffffffff81110880>] ? select_bad_process+0xd0/0x110 2012-01-19 19:11:06 [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0 2012-01-19 19:11:06 [<ffffffff81110b19>] ? out_of_memory+0x199/0x210 2012-01-19 19:11:06 [<ffffffff81120262>] ? __alloc_pages_nodemask+0x812/0x8b0 2012-01-19 19:11:06 [<ffffffff81010e86>] ? dma_generic_alloc_coherent+0xa6/0x160 2012-01-19 19:11:06 [<ffffffffa0017e99>] ? mlx4_create_eq+0x139/0x6b0 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa001857d>] ? mlx4_init_eq_table+0x16d/0x560 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa00185f9>] ? mlx4_init_eq_table+0x1e9/0x560 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa001d4d0>] ? mlx4_setup_hca+0xa0/0x5c0 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa001e045>] ? __mlx4_init_one+0x2f5/0x880 [mlx4_core] 2012-01-19 19:11:06 [<ffffffffa002315f>] ? mlx4_init_one+0x42/0x47 [mlx4_core] 2012-01-19 19:11:06 [<ffffffff81281107>] ? local_pci_probe+0x17/0x20 2012-01-19 19:11:06 [<ffffffff812822f1>] ? pci_device_probe+0x101/0x120 2012-01-19 19:11:06 [<ffffffff8133bb92>] ? driver_sysfs_add+0x62/0x90 2012-01-19 19:11:06 [<ffffffff8133bd30>] ? driver_probe_device+0xa0/0x2a0 2012-01-19 19:11:06 [<ffffffff8133bfdb>] ? __driver_attach+0xab/0xb0 2012-01-19 19:11:06 [<ffffffff8133bf30>] ? __driver_attach+0x0/0xb0 2012-01-19 19:11:06 [<ffffffff8133af94>] ? bus_for_each_dev+0x64/0x90 2012-01-19 19:11:06 [<ffffffff8133bace>] ? driver_attach+0x1e/0x20 2012-01-19 19:11:06 [<ffffffff8133b3d0>] ? bus_add_driver+0x200/0x300 2012-01-19 19:11:06 [<ffffffff8133c306>] ? driver_register+0x76/0x140 2012-01-19 19:11:06 [<ffffffff810899a8>] ? __create_workqueue_key+0x1e8/0x280 2012-01-19 19:11:06 [<ffffffff81282556>] ? __pci_register_driver+0x56/0xd0 2012-01-19 19:11:06 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:07 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:07 [<ffffffffa002e0b2>] ? mlx4_init+0x81/0xbf [mlx4_core] 2012-01-19 19:11:07 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 2012-01-19 19:11:07 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 2012-01-19 19:11:07 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b 2012-01-19 19:11:07 Mem-Info: 2012-01-19 19:11:07 Node 1 DMA per-cpu: 2012-01-19 19:11:07 CPU 0: hi: 0, btch: 1 usd: 0 2012-01-19 19:11:07 Node 1 DMA32 per-cpu: 2012-01-19 19:11:07 CPU 0: hi: 90, btch: 15 usd: 15 2012-01-19 19:11:07 active_anon:1504 inactive_anon:1545 isolated_anon:0 2012-01-19 19:11:07 active_file:0 inactive_file:0 isolated_file:0 2012-01-19 19:11:07 unevictable:6725 dirty:0 writeback:0 unstable:0 2012-01-19 19:11:07 free:533 slab_reclaimable:1170 slab_unreclaimable:4813 2012-01-19 19:11:07 mapped:414 shmem:23 pagetables:882 bounce:0 2012-01-19 19:11:07 Node 1 DMA free:388kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2012-01-19 19:11:07 lowmem_reserve[]: 0 188 188 188 2012-01-19 19:11:07 Node 1 DMA32 free:1744kB min:1752kB low:2188kB high:2628kB active_anon:6016kB inactive_anon:6180kB active_file:0kB inactive_file:0kB unevictable:26900kB isolated(anon):0kB isolated(file):0kB present:193040kB mlocked:0kB dirty:0kB writeback:0kB mapped:1656kB shmem:92kB slab_reclaimable:4680kB slab_unreclaimable:19252kB kernel_stack:1144kB pagetables:3528kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-01-19 19:11:07 lowmem_reserve[]: 0 0 0 0 2012-01-19 19:11:07 Node 1 DMA: 1*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 388kB 2012-01-19 19:11:07 Node 1 DMA32: 36*4kB 11*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1752kB 2012-01-19 19:11:07 6748 total pagecache pages 2012-01-19 19:11:07 0 pages in swap cache 2012-01-19 19:11:07 Swap cache stats: add 0, delete 0, find 0/0 2012-01-19 19:11:07 Free swap = 0kB 2012-01-19 19:11:07 Total swap = 0kB 2012-01-19 19:11:07 57338 pages RAM 2012-01-19 19:11:07 13175 pages reserved 2012-01-19 19:11:07 17977 pages shared 2012-01-19 19:11:07 35299 pages non-shared 2012-01-19 19:11:07 Out of memory: kill process 321 (udevadm) score 132 or a child 2012-01-19 19:11:07 Killed process 321 (udevadm) vsz:8492kB, anon-rss:116kB, file-rss:660kB 2012-01-19 19:11:07 /lib/dracut-lib.modprobe invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=-17 2012-01-19 19:11:07 sh: line 157: modprobe cpuset=/ mems_allowed=1 2012-01-19 19:11:07 321 Killed Pid: 252, comm: modprobe Not tainted 2.6.32-131.17.1.el6_lustre.x86_64 #1 2012-01-19 19:11:07 udevCall Trace: 2012-01-19 19:11:07 adm settle --exi [<ffffffff810c00f1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 2012-01-19 19:11:07 t-if-exists=/ini [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0 2012-01-19 19:11:07 tqueue/work $set [<ffffffff81110880>] ? select_bad_process+0xd0/0x110 2012-01-19 19:11:07 tle_exit_if_exis [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0 2012-01-19 19:11:07 ts 2012-01-19 19:11:07 [<ffffffff81110b19>] ? out_of_memory+0x199/0x210 2012-01-19 19:11:07 [<ffffffff81120262>] ? __alloc_pages_nodemask+0x812/0x8b0 2012-01-19 19:11:07 [<ffffffff81010e86>] ? dma_generic_alloc_coherent+0xa6/0x160 2012-01-19 19:11:07 [<ffffffffa0017e99>] ? mlx4_create_eq+0x139/0x6b0 [mlx4_core] 2012-01-19 19:11:07 [<ffffffffa001857d>] ? mlx4_init_eq_table+0x16d/0x560 [mlx4_core] 2012-01-19 19:11:07 [<ffffffffa00185f9>] ? mlx4_init_eq_table+0x1e9/0x560 [mlx4_core] 2012-01-19 19:11:07 [<ffffffffa001d4d0>] ? mlx4_setup_hca+0xa0/0x5c0 [mlx4_core] 2012-01-19 19:11:07 [<ffffffffa001e045>] ? __mlx4_init_one+0x2f5/0x880 [mlx4_core] 2012-01-19 19:11:07 [<ffffffffa002315f>] ? mlx4_init_one+0x42/0x47 [mlx4_core] 2012-01-19 19:11:07 [<ffffffff81281107>] ? local_pci_probe+0x17/0x20 2012-01-19 19:11:07 [<ffffffff812822f1>] ? pci_device_probe+0x101/0x120 2012-01-19 19:11:07 [<ffffffff8133bb92>] ? driver_sysfs_add+0x62/0x90 2012-01-19 19:11:07 [<ffffffff8133bd30>] ? driver_probe_device+0xa0/0x2a0 2012-01-19 19:11:07 [<ffffffff8133bfdb>] ? __driver_attach+0xab/0xb0 2012-01-19 19:11:07 [<ffffffff8133bf30>] ? __driver_attach+0x0/0xb0 2012-01-19 19:11:07 [<ffffffff8133af94>] ? bus_for_each_dev+0x64/0x90 2012-01-19 19:11:07 [<ffffffff8133bace>] ? driver_attach+0x1e/0x20 2012-01-19 19:11:07 [<ffffffff8133b3d0>] ? bus_add_driver+0x200/0x300 2012-01-19 19:11:07 [<ffffffff8133c306>] ? driver_register+0x76/0x140 2012-01-19 19:11:07 [<ffffffff810899a8>] ? __create_workqueue_key+0x1e8/0x280 2012-01-19 19:11:07 [<ffffffff81282556>] ? __pci_register_driver+0x56/0xd0 2012-01-19 19:11:07 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:07 [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:07 [<ffffffffa002e0b2>] ? mlx4_init+0x81/0xbf [mlx4_core] 2012-01-19 19:11:07 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 2012-01-19 19:11:07 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 2012-01-19 19:11:07 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b 2012-01-19 19:11:07 Mem-Info: 2012-01-19 19:11:07 Node 1 DMA per-cpu: 2012-01-19 19:11:07 CPU 0: hi: 0, btch: 1 usd: 0 2012-01-19 19:11:07 Node 1 DMA32 per-cpu: 2012-01-19 19:11:07 CPU 0: hi: 90, btch: 15 usd: 28 2012-01-19 19:11:07 active_anon:1484 inactive_anon:1545 isolated_anon:0 2012-01-19 19:11:07 active_file:0 inactive_file:0 isolated_file:0 2012-01-19 19:11:07 unevictable:6725 dirty:0 writeback:0 unstable:0 2012-01-19 19:11:07 free:531 slab_reclaimable:1170 slab_unreclaimable:4813 2012-01-19 19:11:07 mapped:386 shmem:23 pagetables:882 bounce:0 2012-01-19 19:11:07 Node 1 DMA free:388kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2012-01-19 19:11:07 lowmem_reserve[]: 0 188 188 188 2012-01-19 19:11:07 Node 1 DMA32 free:1736kB min:1752kB low:2188kB high:2628kB active_anon:5936kB inactive_anon:6180kB active_file:0kB inactive_file:0kB unevictable:26900kB isolated(anon):0kB isolated(file):0kB present:193040kB mlocked:0kB dirty:0kB writeback:0kB mapped:1544kB shmem:92kB slab_reclaimable:4680kB slab_unreclaimable:19252kB kernel_stack:1144kB pagetables:3528kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-01-19 19:11:07 lowmem_reserve[]: 0 0 0 0 2012-01-19 19:11:07 Node 1 DMA: 1*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 388kB 2012-01-19 19:11:07 Node 1 DMA32: 30*4kB 7*8kB 2*16kB 2*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1744kB 2012-01-19 19:11:07 6748 total pagecache pages 2012-01-19 19:11:07 0 pages in swap cache 2012-01-19 19:11:07 Swap cache stats: add 0, delete 0, find 0/0 2012-01-19 19:11:07 Free swap = 0kB 2012-01-19 19:11:07 Total swap = 0kB 2012-01-19 19:11:07 57338 pages RAM 2012-01-19 19:11:07 13175 pages reserved 2012-01-19 19:11:07 17926 pages shared 2012-01-19 19:11:07 35213 pages non-shared 2012-01-19 19:11:07 Out of memory: kill process 322 (init) score 181 or a child 2012-01-19 19:11:07 Killed process 322 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 /lib/dracut-lib.Out of memory: kill process 323 (init) score 181 or a child 2012-01-19 19:11:07 sh: line 73: 3Killed process 323 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 22 Killed Out of memory: kill process 324 (init) score 181 or a child 2012-01-19 19:11:07 ( . "Killed process 324 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 $f" ) 2012-01-19 19:11:07 /init: liOut of memory: kill process 325 (init) score 181 or a child 2012-01-19 19:11:07 ne 221: 323 KiKilled process 325 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 lled Out of memory: kill process 326 (init) score 181 or a child 2012-01-19 19:11:07 $UDEV_QUEUKilled process 326 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 E_EMPTY > /dev/nOut of memory: kill process 327 (init) score 181 or a child 2012-01-19 19:11:07 ull 2>&1 2012-01-19 19:11:07 /lib/dKilled process 327 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 racut-lib.sh: liOut of memory: kill process 328 (init) score 181 or a child 2012-01-19 19:11:07 ne 73: 324 KilKilled process 328 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 led Out of memory: kill process 329 (init) score 181 or a child 2012-01-19 19:11:07 ( . "$f" ) 2012-01-19 19:11:07 Killed process 329 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 2012-01-19 19:11:07 /lib/dracut-libOut of memory: kill process 330 (init) score 181 or a child 2012-01-19 19:11:07 .sh: line 157: Killed process 330 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 325 Killed Out of memory: kill process 331 (init) score 181 or a child 2012-01-19 19:11:07 udeKilled process 331 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 vadm settle --exOut of memory: kill process 332 (init) score 181 or a child 2012-01-19 19:11:07 it-if-exists=/inKilled process 332 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 itqueue/work $seOut of memory: kill process 333 (init) score 181 or a child 2012-01-19 19:11:07 ttle_exit_if_exiKilled process 333 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 sts 2012-01-19 19:11:07 /lib/dracutOut of memory: kill process 334 (init) score 181 or a child 2012-01-19 19:11:07 -lib.sh: line 73Killed process 334 (init) vsz:11624kB, anon-rss:464kB, file-rss:48kB 2012-01-19 19:11:07 : 326 Killed Out of memory: kill process 335 (init) score 181 or a child 2012-01-19 19:11:07 Killed process 335 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 ( . "$f" ) 2012-01-19 19:11:07 /iniOut of memory: kill process 336 (init) score 181 or a child 2012-01-19 19:11:07 t: line 221: 3Killed process 336 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 27 Killed Out of memory: kill process 337 (init) score 181 or a child 2012-01-19 19:11:07 $UDEVKilled process 337 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 _QUEUE_EMPTY > /Out of memory: kill process 338 (init) score 181 or a child 2012-01-19 19:11:07 dev/null 2>&1 2012-01-19 19:11:07 /Killed process 338 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 lib/dracut-lib.sOut of memory: kill process 339 (init) score 181 or a child 2012-01-19 19:11:07 h: line 73: 32Killed process 339 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 8 Killed Out of memory: kill process 340 (init) score 181 or a child 2012-01-19 19:11:07 ( . "$Killed process 340 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 f" ) 2012-01-19 19:11:07 /lib/dracuOut of memory: kill process 341 (init) score 181 or a child 2012-01-19 19:11:07 t-lib.sh: line 1Killed process 341 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 57: 329 KilledOut of memory: kill process 342 (init) score 181 or a child 2012-01-19 19:11:07 Killed process 342 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 udevadm settleOut of memory: kill process 343 (init) score 181 or a child 2012-01-19 19:11:07 --exit-if-existKilled process 343 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 s=/initqueue/worOut of memory: kill process 344 (init) score 181 or a child 2012-01-19 19:11:07 k $settle_exit_iKilled process 344 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 f_exists 2012-01-19 19:11:07 /lib/dOut of memory: kill process 345 (init) score 181 or a child 2012-01-19 19:11:07 racut-lib.sh: liKilled process 345 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 ne 73: 330 KilOut of memory: kill process 346 (init) score 181 or a child 2012-01-19 19:11:07 led Killed process 346 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:07 ( . "$f" ) 2012-01-19 19:11:07 Out of memory: kill process 347 (init) score 181 or a child 2012-01-19 19:11:07 2012-01-19 19:11:07 /init: line 221Killed process 347 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 : 331 Killed Out of memory: kill process 348 (init) score 181 or a child 2012-01-19 19:11:08 Killed process 348 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 $UDEV_QUEUE_EMPTOut of memory: kill process 349 (init) score 181 or a child 2012-01-19 19:11:08 Y > /dev/null 2>Killed process 349 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 &1 2012-01-19 19:11:08 /lib/dracut-Out of memory: kill process 350 (init) score 181 or a child 2012-01-19 19:11:08 lib.sh: line 73:Killed process 350 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 332 Killed Out of memory: kill process 351 (init) score 181 or a child 2012-01-19 19:11:08 (Killed process 351 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 . "$f" ) 2012-01-19 19:11:08 /lib/Out of memory: kill process 352 (init) score 181 or a child 2012-01-19 19:11:08 dracut-lib.sh: lKilled process 352 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 ine 157: 333 KOut of memory: kill process 353 (init) score 181 or a child 2012-01-19 19:11:08 illed Killed process 353 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 udevadm sOut of memory: kill process 354 (init) score 181 or a child 2012-01-19 19:11:08 ettle --exit-if-Killed process 354 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 exists=/initqueuOut of memory: kill process 355 (init) score 181 or a child 2012-01-19 19:11:08 e/work $settle_eKilled process 355 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 xit_if_exists 2012-01-19 19:11:08 /Out of memory: kill process 356 (init) score 181 or a child 2012-01-19 19:11:08 lib/dracut-lib.sKilled process 356 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 h: line 73: 33Out of memory: kill process 357 (init) score 181 or a child 2012-01-19 19:11:08 4 Killed Killed process 357 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 ( . "$Out of memory: kill process 358 (init) score 181 or a child 2012-01-19 19:11:08 f" ) 2012-01-19 19:11:08 /init: linKilled process 358 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 e 221: 335 KilOut of memory: kill process 359 (init) score 181 or a child 2012-01-19 19:11:08 led Killed process 359 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 $UDEV_QUEUEOut of memory: kill process 360 (init) score 181 or a child 2012-01-19 19:11:08 _EMPTY > /dev/nuKilled process 360 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 ll 2>&1 2012-01-19 19:11:08 /lib/drOut of memory: kill process 361 (init) score 181 or a child 2012-01-19 19:11:08 acut-lib.sh: linKilled process 361 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 e 73: 336 KillOut of memory: kill process 362 (init) score 181 or a child 2012-01-19 19:11:08 ed Killed process 362 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 ( . "$f" ) 2012-01-19 19:11:08 Out of memory: kill process 363 (init) score 181 or a child 2012-01-19 19:11:08 /lib/dracut-lib.Killed process 363 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 sh: line 157: Out of memory: kill process 364 (init) score 181 or a child 2012-01-19 19:11:08 337 Killed Killed process 364 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 udevOut of memory: kill process 365 (init) score 181 or a child 2012-01-19 19:11:08 adm settle --exiKilled process 365 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 t-if-exists=/iniOut of memory: kill process 366 (init) score 181 or a child 2012-01-19 19:11:08 tqueue/work $setKilled process 366 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 tle_exit_if_exisOut of memory: kill process 367 (init) score 181 or a child 2012-01-19 19:11:08 ts 2012-01-19 19:11:08 /lib/dracut-Killed process 367 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 lib.sh: line 73:Out of memory: kill process 368 (init) score 181 or a child 2012-01-19 19:11:08 338 Killed Killed process 368 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 (Out of memory: kill process 369 (init) score 181 or a child 2012-01-19 19:11:08 . "$f" ) 2012-01-19 19:11:08 /initKilled process 369 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 : line 221: 33Out of memory: kill process 370 (init) score 181 or a child 2012-01-19 19:11:08 9 Killed Killed process 370 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 $UDEV_Out of memory: kill process 371 (init) score 181 or a child 2012-01-19 19:11:08 QUEUE_EMPTY > /dKilled process 371 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 ev/null 2>&1 2012-01-19 19:11:08 /lOut of memory: kill process 372 (init) score 181 or a child 2012-01-19 19:11:08 ib/dracut-lib.shKilled process 372 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 : line 73: 340Out of memory: kill process 373 (init) score 181 or a child 2012-01-19 19:11:08 Killed Killed process 373 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 ( . "$fOut of memory: kill process 374 (init) score 181 or a child 2012-01-19 19:11:08 " ) 2012-01-19 19:11:08 /lib/dracutKilled process 374 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 -lib.sh: line 15Out of memory: kill process 375 (init) score 181 or a child 2012-01-19 19:11:08 7: 341 Killed Killed process 375 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 Out of memory: kill process 376 (init) score 181 or a child 2012-01-19 19:11:08 udevadm settle Killed process 376 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 --exit-if-existsOut of memory: kill process 377 (init) score 181 or a child 2012-01-19 19:11:08 =/initqueue/workKilled process 377 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 $settle_exit_ifOut of memory: kill process 378 (init) score 181 or a child 2012-01-19 19:11:08 _exists 2012-01-19 19:11:08 /lib/drKilled process 378 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 acut-lib.sh: linOut of memory: kill process 379 (init) score 181 or a child 2012-01-19 19:11:08 e 73: 342 KillKilled process 379 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 ed Out of memory: kill process 380 (init) score 181 or a child 2012-01-19 19:11:08 ( . "$f" ) 2012-01-19 19:11:08 Killed process 380 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 /init: line 221:Out of memory: kill process 381 (init) score 181 or a child 2012-01-19 19:11:08 343 Killed Killed process 381 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 $Out of memory: kill process 382 (init) score 181 or a child 2012-01-19 19:11:08 UDEV_QUEUE_EMPTYKilled process 382 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 > /dev/null 2>&Out of memory: kill process 383 (init) score 181 or a child 2012-01-19 19:11:08 1 2012-01-19 19:11:08 /lib/dracut-lKilled process 383 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 ib.sh: line 73: Out of memory: kill process 384 (init) score 181 or a child 2012-01-19 19:11:08 344 Killed Killed process 384 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 ( Out of memory: kill process 385 (init) score 181 or a child 2012-01-19 19:11:08 . "$f" ) 2012-01-19 19:11:08 /lib/dKilled process 385 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 racut-lib.sh: liOut of memory: kill process 386 (init) score 181 or a child 2012-01-19 19:11:08 ne 157: 345 KiKilled process 386 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 lled Out of memory: kill process 387 (init) score 181 or a child 2012-01-19 19:11:08 udevadm seKilled process 387 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 ttle --exit-if-eOut of memory: kill process 388 (init) score 181 or a child 2012-01-19 19:11:08 xists=/initqueueKilled process 388 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 /work $settle_exOut of memory: kill process 389 (init) score 181 or a child 2012-01-19 19:11:08 it_if_exists 2012-01-19 19:11:08 /lKilled process 389 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 ib/dracut-lib.shOut of memory: kill process 390 (init) score 181 or a child 2012-01-19 19:11:08 : line 73: 346Killed process 390 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 Killed Out of memory: kill process 391 (init) score 181 or a child 2012-01-19 19:11:08 ( . "$fKilled process 391 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 " ) 2012-01-19 19:11:08 /init: lineOut of memory: kill process 392 (init) score 181 or a child 2012-01-19 19:11:08 221: 347 KillKilled process 392 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 ed Out of memory: kill process 393 (init) score 181 or a child 2012-01-19 19:11:08 $UDEV_QUEUE_Killed process 393 (init) vsz:11624kB, anon-rss:464kB, file-rss:44kB 2012-01-19 19:11:08 EMPTY > /dev/nulKernel panic - not syncing: Out of memory and no killable processes... 2012-01-19 19:11:08 2012-01-19 19:11:08 l 2>&1 2012-01-19 19:11:08 /lib/draPid: 252, comm: modprobe Not tainted 2.6.32-131.17.1.el6_lustre.x86_64 #1 2012-01-19 19:11:08 cut-lib.sh: lineCall Trace: 2012-01-19 19:11:08 73: 348 Kille [<ffffffff814dac78>] ? panic+0x78/0x143 2012-01-19 19:11:08 d [<ffffffff81110971>] ? __out_of_memory+0xb1/0xc0 2012-01-19 19:11:08 ( . "$f" ) 2012-01-19 19:11:08 / [<ffffffff81110b19>] ? out_of_memory+0x199/0x210 2012-01-19 19:11:08 lib/dracut-lib.s [<ffffffff81120262>] ? __alloc_pages_nodemask+0x812/0x8b0 2012-01-19 19:11:08 h: line 157: 3 [<ffffffff81010e86>] ? dma_generic_alloc_coherent+0xa6/0x160 2012-01-19 19:11:08 49 Killed [<ffffffffa0017e99>] ? mlx4_create_eq+0x139/0x6b0 [mlx4_core] 2012-01-19 19:11:08 udeva [<ffffffffa001857d>] ? mlx4_init_eq_table+0x16d/0x560 [mlx4_core] 2012-01-19 19:11:08 dm settle --exit [<ffffffffa00185f9>] ? mlx4_init_eq_table+0x1e9/0x560 [mlx4_core] 2012-01-19 19:11:08 -if-exists=/init [<ffffffffa001d4d0>] ? mlx4_setup_hca+0xa0/0x5c0 [mlx4_core] 2012-01-19 19:11:08 queue/work $sett [<ffffffffa001e045>] ? __mlx4_init_one+0x2f5/0x880 [mlx4_core] 2012-01-19 19:11:08 le_exit_if_exist [<ffffffffa002315f>] ? mlx4_init_one+0x42/0x47 [mlx4_core] 2012-01-19 19:11:08 s 2012-01-19 19:11:08 /lib/dracut-l [<ffffffff81281107>] ? local_pci_probe+0x17/0x20 2012-01-19 19:11:08 ib.sh: line 73: [<ffffffff812822f1>] ? pci_device_probe+0x101/0x120 2012-01-19 19:11:08 350 Killed [<ffffffff8133bb92>] ? driver_sysfs_add+0x62/0x90 2012-01-19 19:11:08 ( [<ffffffff8133bd30>] ? driver_probe_device+0xa0/0x2a0 2012-01-19 19:11:08 . "$f" ) 2012-01-19 19:11:08 /init: [<ffffffff8133bfdb>] ? __driver_attach+0xab/0xb0 2012-01-19 19:11:08 line 221: 351 [<ffffffff8133bf30>] ? __driver_attach+0x0/0xb0 2012-01-19 19:11:08 Killed [<ffffffff8133af94>] ? bus_for_each_dev+0x64/0x90 2012-01-19 19:11:08 $UDEV_Q [<ffffffff8133bace>] ? driver_attach+0x1e/0x20 2012-01-19 19:11:08 UEUE_EMPTY > /de [<ffffffff8133b3d0>] ? bus_add_driver+0x200/0x300 2012-01-19 19:11:08 v/null 2>&1 2012-01-19 19:11:08 /li [<ffffffff8133c306>] ? driver_register+0x76/0x140 2012-01-19 19:11:08 b/dracut-lib.sh: [<ffffffff810899a8>] ? __create_workqueue_key+0x1e8/0x280 2012-01-19 19:11:08 line 73: 352 [<ffffffff81282556>] ? __pci_register_driver+0x56/0xd0 2012-01-19 19:11:08 Killed [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:08 ( . "$f" [<ffffffffa002e031>] ? mlx4_init+0x0/0xbf [mlx4_core] 2012-01-19 19:11:08 ) 2012-01-19 19:11:08 /lib/dracut- [<ffffffffa002e0b2>] ? mlx4_init+0x81/0xbf [mlx4_core] 2012-01-19 19:11:08 lib.sh: line 157 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 2012-01-19 19:11:08 : 353 Killed [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 2012-01-19 19:11:08 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b 2012-01-19 19:11:08 udevadm settle - <ConMan> Console [hyperion-rst6] departed by <root@localhost> on pts/107 at 01-19 19:57.