Console [seqlac2] log at 2012-11-19 10:00:00 PST. 2012-11-19 10:02:18 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:02:18 Call Trace: 2012-11-19 10:02:18 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:02:18 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:02:18 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:02:18 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:02:18 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:02:18 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:02:18 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:02:18 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:02:18 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:02:18 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:02:18 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:02:18 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:02:18 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:02:18 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:02:18 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:02:18 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:02:18 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:02:18 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:02:18 --- Exception: 501 at .pseries_dedicated_idle_sleep+0x84/0x1f0 2012-11-19 10:02:18 LR = .pseries_dedicated_idle_sleep+0x7c/0x1f0 2012-11-19 10:02:18 [c000000f59b7bd80] [0001d66ca6e0902a] 0x1d66ca6e0902a (unreliable) 2012-11-19 10:02:18 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 10:02:18 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:02:18 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:02:18 Mem-Info: 2012-11-19 10:02:18 Node 0 DMA per-cpu: 2012-11-19 10:02:18 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 4: hi: 6, btch: 1 usd: 3 2012-11-19 10:02:18 CPU 5: hi: 6, btch: 1 usd: 2 2012-11-19 10:02:18 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 9: hi: 6, btch: 1 usd: 2 2012-11-19 10:02:18 CPU 10: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 21: hi: 6, btch: 1 usd: 2 2012-11-19 10:02:18 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 24: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:18 CPU 25: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:18 CPU 26: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:18 CPU 27: hi: 6, btch: 1 usd: 4 2012-11-19 10:02:18 CPU 28: hi: 6, btch: 1 usd: 1 2012-11-19 10:02:18 CPU 29: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:18 CPU 30: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:18 CPU 31: hi: 6, btch: 1 usd: 3 2012-11-19 10:02:18 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 33: hi: 6, btch: 1 usd: 3 2012-11-19 10:02:18 CPU 34: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 35: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 36: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:18 CPU 37: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:18 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:18 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:18 CPU 40: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 41: hi: 6, btch: 1 usd: 3 2012-11-19 10:02:18 CPU 42: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:18 CPU 44: hi: 6, btch: 1 usd: 1 2012-11-19 10:02:18 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:18 CPU 46: hi: 6, btch: 1 usd: 2 2012-11-19 10:02:18 CPU 47: hi: 6, btch: 1 usd: 2 2012-11-19 10:02:18 active_anon:4223 inactive_anon:4519 isolated_anon:0 2012-11-19 10:02:18 active_file:91839 inactive_file:816404 isolated_file:0 2012-11-19 10:02:18 unevictable:0 dirty:400 writeback:48 unstable:0 2012-11-19 10:02:18 free:800 slab_reclaimable:29691 slab_unreclaimable:25261 2012-11-19 10:02:18 mapped:1519 shmem:77 pagetables:544 bounce:0 2012-11-19 10:02:18 Node 0 DMA free:51200kB min:32128kB low:40128kB high:48192kB active_anon:270272kB inactive_anon:289216kB active_file:5877696kB inactive_file:52249856kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:25600kB writeback:3072kB mapped:97216kB shmem:4928kB slab_reclaimable:1900224kB slab_unreclaimable:1616704kB kernel_stack:26704kB pagetables:34816kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:02:18 lowmem_reserve[]: 0 0 0 2012-11-19 10:02:18 Node 0 DMA: 829*64kB 6*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 57152kB 2012-11-19 10:02:18 908226 total pagecache pages 2012-11-19 10:02:18 0 pages in swap cache 2012-11-19 10:02:18 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:02:18 Free swap = 4194176kB 2012-11-19 10:02:18 Total swap = 4194176kB 2012-11-19 10:02:18 1011712 pages RAM 2012-11-19 10:02:18 20742 pages reserved 2012-11-19 10:02:18 59138 pages shared 2012-11-19 10:02:18 938329 pages non-shared 2012-11-19 10:02:40 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:02:40 Call Trace: 2012-11-19 10:02:40 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:02:40 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:02:40 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:02:40 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:02:40 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:02:40 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:02:40 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:02:40 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:02:40 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:02:40 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:02:40 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:02:40 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:02:40 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:02:40 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:02:40 [c000000f59b7b970] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:02:40 [c000000f59b7ba10] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:02:40 [c000000f59b7ba90] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:02:40 --- Exception: 501 at 0x3af096d4c9f4 2012-11-19 10:02:40 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 10:02:40 [c000000f59b7bb40] [c000000000004898] hardware_interrupt_entry+0x18/0x80 (unreliable) 2012-11-19 10:02:40 --- Exception: 501 at .raw_local_irq_restore+0x34/0x60 2012-11-19 10:02:40 LR = .cpu_idle+0x14c/0x1d0 2012-11-19 10:02:40 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 (unreliable) 2012-11-19 10:02:40 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:02:40 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:02:40 Mem-Info: 2012-11-19 10:02:40 Node 0 DMA per-cpu: 2012-11-19 10:02:40 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 1: hi: 6, btch: 1 usd: 2 2012-11-19 10:02:40 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 4: hi: 6, btch: 1 usd: 1 2012-11-19 10:02:40 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:40 CPU 10: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 11: hi: 6, btch: 1 usd: 4 2012-11-19 10:02:40 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 16: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:40 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 24: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:40 CPU 25: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:40 CPU 26: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:40 CPU 27: hi: 6, btch: 1 usd: 4 2012-11-19 10:02:40 CPU 28: hi: 6, btch: 1 usd: 4 2012-11-19 10:02:40 CPU 29: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 30: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 31: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 33: hi: 6, btch: 1 usd: 4 2012-11-19 10:02:40 CPU 34: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 35: hi: 6, btch: 1 usd: 4 2012-11-19 10:02:40 CPU 36: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 37: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:40 CPU 39: hi: 6, btch: 1 usd: 2 2012-11-19 10:02:40 CPU 40: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 41: hi: 6, btch: 1 usd: 2 2012-11-19 10:02:40 CPU 42: hi: 6, btch: 1 usd: 4 2012-11-19 10:02:40 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 10:02:40 CPU 44: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 10:02:40 CPU 46: hi: 6, btch: 1 usd: 4 2012-11-19 10:02:40 CPU 47: hi: 6, btch: 1 usd: 3 2012-11-19 10:02:40 active_anon:2988 inactive_anon:4519 isolated_anon:0 2012-11-19 10:02:40 active_file:91154 inactive_file:818469 isolated_file:33 2012-11-19 10:02:40 unevictable:0 dirty:264 writeback:66 unstable:0 2012-11-19 10:02:40 free:996 slab_reclaimable:29406 slab_unreclaimable:25175 2012-11-19 10:02:40 mapped:1549 shmem:77 pagetables:523 bounce:0 2012-11-19 10:02:40 Node 0 DMA free:64512kB min:32128kB low:40128kB high:48192kB active_anon:191232kB inactive_anon:289216kB active_file:5833856kB inactive_file:52382016kB unevictable:0kB isolated(anon):0kB isolated(file):2112kB present:64694208kB mlocked:0kB dirty:16896kB writeback:4224kB mapped:99136kB shmem:4928kB slab_reclaimable:1881984kB slab_unreclaimable:1611200kB kernel_stack:26640kB pagetables:33472kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:02:40 lowmem_reserve[]: 0 0 0 2012-11-19 10:02:40 Node 0 DMA: 873*64kB 41*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 64448kB 2012-11-19 10:02:40 908490 total pagecache pages 2012-11-19 10:02:40 0 pages in swap cache 2012-11-19 10:02:40 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:02:40 Free swap = 4194176kB 2012-11-19 10:02:40 Total swap = 4194176kB 2012-11-19 10:02:40 1011712 pages RAM 2012-11-19 10:02:40 20742 pages reserved 2012-11-19 10:02:40 47372 pages shared 2012-11-19 10:02:40 949794 pages non-shared 2012-11-19 10:16:23 rpciod/8: page allocation failure. order:1, mode:0x20 2012-11-19 10:16:23 Call Trace: 2012-11-19 10:16:23 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:16:23 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:16:23 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:16:23 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:16:23 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:16:23 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:16:23 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:16:23 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:16:23 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:16:23 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:16:23 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:16:23 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:16:23 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:16:23 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:16:23 [c000000f498c6920] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:16:23 [c000000f498c69c0] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:16:23 [c000000f498c6a40] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:16:23 [c000000f498c6af0] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:16:23 --- Exception: 501 at .memcpy+0x34/0x280 2012-11-19 10:16:23 LR = .pskb_expand_head+0x94/0x280 2012-11-19 10:16:23 [c000000f498c6de0] [c0000000004d6b00] .pskb_expand_head+0x70/0x280 (unreliable) 2012-11-19 10:16:23 [c000000f498c6e90] [c0000000004d7358] .__pskb_pull_tail+0x148/0x420 2012-11-19 10:16:23 [c000000f498c6f40] [c0000000004e6e3c] .dev_queue_xmit+0x62c/0x8e0 2012-11-19 10:16:23 [c000000f498c6ff0] [c0000000004efc70] .neigh_connected_output+0x110/0x160 2012-11-19 10:16:23 [c000000f498c70a0] [c00000000053079c] .ip_finish_output+0x6c/0x3e0 2012-11-19 10:16:23 [c000000f498c7140] [c000000000530fa4] .ip_local_out+0x44/0x60 2012-11-19 10:16:23 [c000000f498c71c0] [c0000000005314fc] .ip_queue_xmit+0x16c/0x410 2012-11-19 10:16:23 [c000000f498c72c0] [c000000000549fd4] .tcp_transmit_skb+0x474/0x9b0 2012-11-19 10:16:23 [c000000f498c73c0] [c00000000054d020] .tcp_write_xmit+0x230/0xbb0 2012-11-19 10:16:23 [c000000f498c74d0] [c00000000054da20] .__tcp_push_pending_frames+0x40/0x100 2012-11-19 10:16:23 [c000000f498c7550] [c00000000053d228] .tcp_sendmsg+0x2a8/0xdf0 2012-11-19 10:16:23 [c000000f498c76a0] [c0000000004ca3ec] .sock_sendmsg+0x14c/0x1a0 2012-11-19 10:16:23 [c000000f498c78c0] [c0000000004ca478] .kernel_sendmsg+0x38/0x60 2012-11-19 10:16:23 [c000000f498c7940] [d00000000bfe8ac4] .xs_send_kvec+0xb4/0xd0 [sunrpc] 2012-11-19 10:16:23 [c000000f498c7a00] [d00000000bfe8b9c] .xs_sendpages+0xbc/0x350 [sunrpc] 2012-11-19 10:16:23 [c000000f498c7ad0] [d00000000bfe8fdc] .xs_tcp_send_request+0x6c/0x1e0 [sunrpc] 2012-11-19 10:16:23 [c000000f498c7b80] [d00000000bfe59a4] .xprt_transmit+0xa4/0x340 [sunrpc] 2012-11-19 10:16:23 [c000000f498c7c40] [d00000000bfe1928] .call_transmit+0x1e8/0x2c0 [sunrpc] 2012-11-19 10:16:23 [c000000f498c7cd0] [d00000000bfef154] .__rpc_execute+0xc4/0x470 [sunrpc] 2012-11-19 10:16:23 [c000000f498c7db0] [c0000000000b6cac] .worker_thread+0x1dc/0x3c0 2012-11-19 10:16:23 [c000000f498c7ed0] [c0000000000bd02c] .kthread+0xbc/0xd0 2012-11-19 10:16:23 [c000000f498c7f90] [c000000000032054] .kernel_thread+0x54/0x70 2012-11-19 10:16:23 Mem-Info: 2012-11-19 10:16:23 Node 0 DMA per-cpu: 2012-11-19 10:16:23 CPU 0: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 1: hi: 6, btch: 1 usd: 4 2012-11-19 10:16:23 CPU 2: hi: 6, btch: 1 usd: 4 2012-11-19 10:16:23 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 9: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 10: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 13: hi: 6, btch: 1 usd: 1 2012-11-19 10:16:23 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 18: hi: 6, btch: 1 usd: 4 2012-11-19 10:16:23 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 21: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 24: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 25: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 26: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 29: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 30: hi: 6, btch: 1 usd: 1 2012-11-19 10:16:23 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 32: hi: 6, btch: 1 usd: 1 2012-11-19 10:16:23 CPU 33: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 34: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 35: hi: 6, btch: 1 usd: 4 2012-11-19 10:16:23 CPU 36: hi: 6, btch: 1 usd: 2 2012-11-19 10:16:23 CPU 37: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 39: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 40: hi: 6, btch: 1 usd: 1 2012-11-19 10:16:23 CPU 41: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 42: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 43: hi: 6, btch: 1 usd: 2 2012-11-19 10:16:23 CPU 44: hi: 6, btch: 1 usd: 5 2012-11-19 10:16:23 CPU 45: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 46: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 CPU 47: hi: 6, btch: 1 usd: 0 2012-11-19 10:16:23 active_anon:103347 inactive_anon:4519 isolated_anon:0 2012-11-19 10:16:23 active_file:78044 inactive_file:744663 isolated_file:0 2012-11-19 10:16:23 unevictable:0 dirty:128 writeback:50 unstable:0 2012-11-19 10:16:23 free:935 slab_reclaimable:21606 slab_unreclaimable:19316 2012-11-19 10:16:23 mapped:1679 shmem:77 pagetables:576 bounce:0 2012-11-19 10:16:23 Node 0 DMA free:59840kB min:32128kB low:40128kB high:48192kB active_anon:6614208kB inactive_anon:289216kB active_file:4994816kB inactive_file:47658432kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:8192kB writeback:3200kB mapped:107456kB shmem:4928kB slab_reclaimable:1382784kB slab_unreclaimable:1236224kB kernel_stack:26656kB pagetables:36864kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:16:23 lowmem_reserve[]: 0 0 0 2012-11-19 10:16:23 Node 0 DMA: 345*64kB 83*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 32704kB 2012-11-19 10:16:23 822864 total pagecache pages 2012-11-19 10:16:23 0 pages in swap cache 2012-11-19 10:16:23 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:16:23 Free swap = 4194176kB 2012-11-19 10:16:23 Total swap = 4194176kB 2012-11-19 10:16:23 1011712 pages RAM 2012-11-19 10:16:23 20742 pages reserved 2012-11-19 10:16:23 27965 pages shared 2012-11-19 10:16:23 972293 pages non-shared 2012-11-19 10:28:21 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:28:21 Call Trace: 2012-11-19 10:28:21 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:28:21 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:28:21 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:28:21 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:28:21 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:28:21 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:28:21 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:28:21 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:28:21 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:28:21 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:28:21 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:28:21 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:28:21 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:28:21 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:28:21 [c000000f59b7b970] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:28:21 [c000000f59b7ba10] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:28:21 [c000000f59b7ba90] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:28:21 --- Exception: 501 at 0x3b1784b7b748 2012-11-19 10:28:21 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 10:28:21 [c000000f59b7bb40] [c000000000004898] hardware_interrupt_entry+0x18/0x80 (unreliable) 2012-11-19 10:28:21 --- Exception: 501 at .raw_local_irq_restore+0x34/0x60 2012-11-19 10:28:21 LR = .cpu_idle+0x14c/0x1d0 2012-11-19 10:28:21 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 (unreliable) 2012-11-19 10:28:21 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:28:21 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:28:21 Mem-Info: 2012-11-19 10:28:21 Node 0 DMA per-cpu: 2012-11-19 10:28:21 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 5: hi: 6, btch: 1 usd: 2 2012-11-19 10:28:21 CPU 6: hi: 6, btch: 1 usd: 2 2012-11-19 10:28:21 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:21 CPU 10: hi: 6, btch: 1 usd: 4 2012-11-19 10:28:21 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 12: hi: 6, btch: 1 usd: 3 2012-11-19 10:28:21 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 15: hi: 6, btch: 1 usd: 3 2012-11-19 10:28:21 CPU 16: hi: 6, btch: 1 usd: 2 2012-11-19 10:28:21 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 24: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 25: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 26: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:21 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:21 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:21 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 10:28:21 CPU 30: hi: 6, btch: 1 usd: 3 2012-11-19 10:28:21 CPU 31: hi: 6, btch: 1 usd: 4 2012-11-19 10:28:21 CPU 32: hi: 6, btch: 1 usd: 3 2012-11-19 10:28:21 CPU 33: hi: 6, btch: 1 usd: 3 2012-11-19 10:28:21 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:21 CPU 35: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:21 CPU 36: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 37: hi: 6, btch: 1 usd: 3 2012-11-19 10:28:21 CPU 38: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 39: hi: 6, btch: 1 usd: 2 2012-11-19 10:28:21 CPU 40: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 42: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 44: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 CPU 46: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:21 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:21 active_anon:2869 inactive_anon:5032 isolated_anon:0 2012-11-19 10:28:21 active_file:73559 inactive_file:849707 isolated_file:0 2012-11-19 10:28:21 unevictable:0 dirty:27 writeback:0 unstable:0 2012-11-19 10:28:21 free:3730 slab_reclaimable:18864 slab_unreclaimable:19003 2012-11-19 10:28:21 mapped:1542 shmem:78 pagetables:563 bounce:0 2012-11-19 10:28:21 Node 0 DMA free:239296kB min:32128kB low:40128kB high:48192kB active_anon:183616kB inactive_anon:322048kB active_file:4707776kB inactive_file:54379136kB unevictable:0kB isolated(anon):0kB isolated(file):2112kB present:64694208kB mlocked:0kB dirty:1728kB writeback:0kB mapped:98688kB shmem:4992kB slab_reclaimable:1207296kB slab_unreclaimable:1216192kB kernel_stack:26784kB pagetables:36032kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:28:21 lowmem_reserve[]: 0 0 0 2012-11-19 10:28:21 Node 0 DMA: 3492*64kB 126*128kB 4*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 241152kB 2012-11-19 10:28:21 921552 total pagecache pages 2012-11-19 10:28:21 0 pages in swap cache 2012-11-19 10:28:21 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:28:21 Free swap = 4194176kB 2012-11-19 10:28:21 Total swap = 4194176kB 2012-11-19 10:28:21 1011712 pages RAM 2012-11-19 10:28:21 20742 pages reserved 2012-11-19 10:28:21 18383 pages shared 2012-11-19 10:28:21 976619 pages non-shared 2012-11-19 10:28:37 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:28:37 Call Trace: 2012-11-19 10:28:37 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:28:37 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:28:37 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:28:37 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:28:37 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:28:37 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:28:37 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:28:37 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:28:37 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:28:37 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:28:37 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:28:37 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:28:37 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:28:37 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:28:37 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:28:37 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:28:37 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:28:37 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:28:37 --- Exception: 501 at .ppc64_runlatch_off+0x24/0x60 2012-11-19 10:28:37 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 10:28:37 [c000000f59b7bd80] [0001d7dc38045466] 0x1d7dc38045466 (unreliable) 2012-11-19 10:28:37 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 10:28:37 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:28:37 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:28:37 Mem-Info: 2012-11-19 10:28:37 Node 0 DMA per-cpu: 2012-11-19 10:28:37 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 1: hi: 6, btch: 1 usd: 6 2012-11-19 10:28:37 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 6: hi: 6, btch: 1 usd: 3 2012-11-19 10:28:37 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:37 CPU 10: hi: 6, btch: 1 usd: 3 2012-11-19 10:28:37 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 12: hi: 6, btch: 1 usd: 3 2012-11-19 10:28:37 CPU 13: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:37 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 15: hi: 6, btch: 1 usd: 3 2012-11-19 10:28:37 CPU 16: hi: 6, btch: 1 usd: 3 2012-11-19 10:28:37 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 24: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 25: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 26: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 27: hi: 6, btch: 1 usd: 2 2012-11-19 10:28:37 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:37 CPU 29: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:37 CPU 30: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:37 CPU 31: hi: 6, btch: 1 usd: 2 2012-11-19 10:28:37 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 33: hi: 6, btch: 1 usd: 3 2012-11-19 10:28:37 CPU 34: hi: 6, btch: 1 usd: 4 2012-11-19 10:28:37 CPU 35: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:37 CPU 36: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 37: hi: 6, btch: 1 usd: 2 2012-11-19 10:28:37 CPU 38: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:37 CPU 40: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 41: hi: 6, btch: 1 usd: 4 2012-11-19 10:28:37 CPU 42: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 44: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 CPU 45: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:37 CPU 46: hi: 6, btch: 1 usd: 0 2012-11-19 10:28:37 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 10:28:37 active_anon:3063 inactive_anon:5032 isolated_anon:0 2012-11-19 10:28:37 active_file:73470 inactive_file:852422 isolated_file:33 2012-11-19 10:28:37 unevictable:0 dirty:536 writeback:0 unstable:0 2012-11-19 10:28:37 free:1048 slab_reclaimable:18750 slab_unreclaimable:18974 2012-11-19 10:28:37 mapped:1552 shmem:78 pagetables:567 bounce:0 2012-11-19 10:28:37 Node 0 DMA free:67648kB min:32128kB low:40128kB high:48192kB active_anon:196032kB inactive_anon:322048kB active_file:4702080kB inactive_file:54555008kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:34304kB writeback:0kB mapped:99328kB shmem:4992kB slab_reclaimable:1200000kB slab_unreclaimable:1214336kB kernel_stack:26848kB pagetables:36288kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:28:37 lowmem_reserve[]: 0 0 0 2012-11-19 10:28:37 Node 0 DMA: 709*64kB 146*128kB 4*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 65600kB 2012-11-19 10:28:37 924203 total pagecache pages 2012-11-19 10:28:37 0 pages in swap cache 2012-11-19 10:28:37 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:28:37 Free swap = 4194176kB 2012-11-19 10:28:37 Total swap = 4194176kB 2012-11-19 10:28:37 1011712 pages RAM 2012-11-19 10:28:37 20742 pages reserved 2012-11-19 10:28:37 20699 pages shared 2012-11-19 10:28:37 978212 pages non-shared 2012-11-19 10:29:20 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:29:20 Call Trace: 2012-11-19 10:29:20 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:29:20 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:29:20 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:29:20 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:29:20 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:29:20 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:29:20 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:29:20 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:29:20 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:29:20 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:29:20 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:29:20 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:29:20 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:29:20 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:29:20 [c000000f59b7b970] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:29:20 [c000000f59b7ba10] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:29:20 [c000000f59b7ba90] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:29:20 [c000000f59b7bb40] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:29:20 --- Exception: 501 at .raw_local_irq_restore+0x34/0x60 2012-11-19 10:29:20 LR = .cpu_idle+0x14c/0x1d0 2012-11-19 10:29:20 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 (unreliable) 2012-11-19 10:29:20 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:29:20 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:29:20 Mem-Info: 2012-11-19 10:29:20 Node 0 DMA per-cpu: 2012-11-19 10:29:20 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 6: hi: 6, btch: 1 usd: 4 2012-11-19 10:29:20 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 10:29:20 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 10:29:20 CPU 11: hi: 6, btch: 1 usd: 4 2012-11-19 10:29:20 CPU 12: hi: 6, btch: 1 usd: 3 2012-11-19 10:29:20 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 24: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 25: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 26: hi: 6, btch: 1 usd: 0 2012-11-19 10:29:20 CPU 27: hi: 6, btch: 1 usd: 3 2012-11-19 10:29:20 CPU 28: hi: 6, btch: 1 usd: 1 2012-11-19 10:29:20 CPU 29: hi: 6, btch: 1 usd: 1 2012-11-19 10:29:20 CPU 30: hi: 6, btch: 1 usd: 4 2012-11-19 10:29:20 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 10:29:20 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 10:29:20 CPU 33: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 34: hi: 6, btch: 1 usd: 3 2012-11-19 10:29:20 CPU 35: hi: 6, btch: 1 usd: 4 2012-11-19 10:29:20 CPU 36: hi: 6, btch: 1 usd: 4 2012-11-19 10:29:20 CPU 37: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 38: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 39: hi: 6, btch: 1 usd: 3 2012-11-19 10:29:20 CPU 40: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 41: hi: 6, btch: 1 usd: 0 2012-11-19 10:29:20 CPU 42: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 44: hi: 6, btch: 1 usd: 4 2012-11-19 10:29:20 CPU 45: hi: 6, btch: 1 usd: 0 2012-11-19 10:29:20 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 10:29:20 CPU 47: hi: 6, btch: 1 usd: 3 2012-11-19 10:29:20 active_anon:2870 inactive_anon:5032 isolated_anon:0 2012-11-19 10:29:20 active_file:73381 inactive_file:853222 isolated_file:33 2012-11-19 10:29:20 unevictable:0 dirty:93 writeback:0 unstable:0 2012-11-19 10:29:20 free:1038 slab_reclaimable:18294 slab_unreclaimable:18853 2012-11-19 10:29:20 mapped:1542 shmem:78 pagetables:563 bounce:0 2012-11-19 10:29:20 Node 0 DMA free:67200kB min:32128kB low:40128kB high:48192kB active_anon:183680kB inactive_anon:322048kB active_file:4696384kB inactive_file:54606208kB unevictable:0kB isolated(anon):0kB isolated(file):2112kB present:64694208kB mlocked:0kB dirty:5952kB writeback:0kB mapped:98688kB shmem:4992kB slab_reclaimable:1170816kB slab_unreclaimable:1206592kB kernel_stack:26784kB pagetables:36032kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:29:20 lowmem_reserve[]: 0 0 0 2012-11-19 10:29:20 Node 0 DMA: 790*64kB 130*128kB 4*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 68736kB 2012-11-19 10:29:20 925794 total pagecache pages 2012-11-19 10:29:20 0 pages in swap cache 2012-11-19 10:29:20 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:29:20 Free swap = 4194176kB 2012-11-19 10:29:20 Total swap = 4194176kB 2012-11-19 10:29:20 1011712 pages RAM 2012-11-19 10:29:20 20742 pages reserved 2012-11-19 10:29:20 20861 pages shared 2012-11-19 10:29:20 977620 pages non-shared 2012-11-19 10:32:56 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:32:56 Call Trace: 2012-11-19 10:32:56 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:32:56 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:32:56 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:32:56 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:32:56 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:32:56 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:32:56 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:32:56 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:32:56 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:32:56 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:32:56 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:32:56 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:32:56 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:32:56 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:32:56 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:32:56 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:32:56 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:32:56 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:32:56 --- Exception: 501 at .pseries_dedicated_idle_sleep+0x11c/0x1f0 2012-11-19 10:32:56 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 10:32:56 [c000000f59b7bd80] [0001d818a21bbf80] 0x1d818a21bbf80 (unreliable) 2012-11-19 10:32:56 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 10:32:56 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:32:56 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:32:56 Mem-Info: 2012-11-19 10:32:56 Node 0 DMA per-cpu: 2012-11-19 10:32:56 CPU 0: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 4: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 6: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 9: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 13: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 14: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 24: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 25: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 26: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 30: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 35: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 36: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 37: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 39: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 42: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 44: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 45: hi: 6, btch: 1 usd: 3 2012-11-19 10:32:56 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 active_anon:11878 inactive_anon:5032 isolated_anon:0 2012-11-19 10:32:56 active_file:70681 inactive_file:847921 isolated_file:0 2012-11-19 10:32:56 unevictable:0 dirty:496 writeback:0 unstable:0 2012-11-19 10:32:56 free:1026 slab_reclaimable:16992 slab_unreclaimable:18990 2012-11-19 10:32:56 mapped:1641 shmem:78 pagetables:649 bounce:0 2012-11-19 10:32:56 Node 0 DMA free:65472kB min:32128kB low:40128kB high:48192kB active_anon:760192kB inactive_anon:322048kB active_file:4523584kB inactive_file:54266944kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:31744kB writeback:0kB mapped:105024kB shmem:4992kB slab_reclaimable:1087488kB slab_unreclaimable:1215360kB kernel_stack:27424kB pagetables:41536kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:32:56 lowmem_reserve[]: 0 0 0 2012-11-19 10:32:56 Node 0 DMA: 792*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 50688kB 2012-11-19 10:32:56 918215 total pagecache pages 2012-11-19 10:32:56 0 pages in swap cache 2012-11-19 10:32:56 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:32:56 Free swap = 4194176kB 2012-11-19 10:32:56 Total swap = 4194176kB 2012-11-19 10:32:56 1011712 pages RAM 2012-11-19 10:32:56 20742 pages reserved 2012-11-19 10:32:56 20357 pages shared 2012-11-19 10:32:56 980017 pages non-shared 2012-11-19 10:32:56 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:32:56 Call Trace: 2012-11-19 10:32:56 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:32:56 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:32:56 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:32:56 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:32:56 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:32:56 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:32:56 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:32:56 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:32:56 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:32:56 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:32:56 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:32:56 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:32:56 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:32:56 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:32:56 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:32:56 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:32:56 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:32:56 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:32:56 --- Exception: 501 at .pseries_dedicated_idle_sleep+0x11c/0x1f0 2012-11-19 10:32:56 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 10:32:56 [c000000f59b7bd80] [0001d818a21bbf80] 0x1d818a21bbf80 (unreliable) 2012-11-19 10:32:56 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 10:32:56 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:32:56 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:32:56 Mem-Info: 2012-11-19 10:32:56 Node 0 DMA per-cpu: 2012-11-19 10:32:56 CPU 0: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 4: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 6: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 9: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 13: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 14: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 24: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 25: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 26: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 30: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 35: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 36: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 37: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 39: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 40: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 42: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 44: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 45: hi: 6, btch: 1 usd: 3 2012-11-19 10:32:56 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 active_anon:12046 inactive_anon:5032 isolated_anon:0 2012-11-19 10:32:56 active_file:70681 inactive_file:847936 isolated_file:0 2012-11-19 10:32:56 unevictable:0 dirty:504 writeback:0 unstable:0 2012-11-19 10:32:56 free:843 slab_reclaimable:16992 slab_unreclaimable:18990 2012-11-19 10:32:56 mapped:1641 shmem:78 pagetables:649 bounce:0 2012-11-19 10:32:56 Node 0 DMA free:53952kB min:32128kB low:40128kB high:48192kB active_anon:770944kB inactive_anon:322048kB active_file:4523584kB inactive_file:54267904kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:32256kB writeback:0kB mapped:105024kB shmem:4992kB slab_reclaimable:1087488kB slab_unreclaimable:1215360kB kernel_stack:27424kB pagetables:41536kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:32:56 lowmem_reserve[]: 0 0 0 2012-11-19 10:32:56 Node 0 DMA: 607*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 38848kB 2012-11-19 10:32:56 918239 total pagecache pages 2012-11-19 10:32:56 0 pages in swap cache 2012-11-19 10:32:56 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:32:56 Free swap = 4194176kB 2012-11-19 10:32:56 Total swap = 4194176kB 2012-11-19 10:32:56 1011712 pages RAM 2012-11-19 10:32:56 20742 pages reserved 2012-11-19 10:32:56 20355 pages shared 2012-11-19 10:32:56 980145 pages non-shared 2012-11-19 10:32:56 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:32:56 Call Trace: 2012-11-19 10:32:56 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:32:56 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:32:56 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:32:56 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:32:56 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:32:56 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:32:56 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:32:56 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:32:56 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:32:56 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:32:56 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:32:56 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:32:56 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:32:56 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:32:56 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:32:56 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:32:56 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:32:56 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:32:56 --- Exception: 501 at .pseries_dedicated_idle_sleep+0x11c/0x1f0 2012-11-19 10:32:56 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 10:32:56 [c000000f59b7bd80] [0001d818a21bbf80] 0x1d818a21bbf80 (unreliable) 2012-11-19 10:32:56 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 10:32:56 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:32:56 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:32:56 Mem-Info: 2012-11-19 10:32:56 Node 0 DMA per-cpu: 2012-11-19 10:32:56 CPU 0: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 4: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 6: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 9: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 13: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 14: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 24: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 25: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 26: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 30: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 35: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 36: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 37: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 39: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 40: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 42: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 44: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 45: hi: 6, btch: 1 usd: 3 2012-11-19 10:32:56 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 active_anon:12169 inactive_anon:5032 isolated_anon:0 2012-11-19 10:32:56 active_file:70681 inactive_file:848020 isolated_file:0 2012-11-19 10:32:56 unevictable:0 dirty:590 writeback:0 unstable:0 2012-11-19 10:32:56 free:646 slab_reclaimable:16992 slab_unreclaimable:18993 2012-11-19 10:32:56 mapped:1641 shmem:78 pagetables:651 bounce:0 2012-11-19 10:32:56 Node 0 DMA free:41344kB min:32128kB low:40128kB high:48192kB active_anon:778816kB inactive_anon:322048kB active_file:4523584kB inactive_file:54273280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:37760kB writeback:0kB mapped:105024kB shmem:4992kB slab_reclaimable:1087488kB slab_unreclaimable:1215552kB kernel_stack:27424kB pagetables:41664kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:32:56 lowmem_reserve[]: 0 0 0 2012-11-19 10:32:56 Node 0 DMA: 527*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 33728kB 2012-11-19 10:32:56 918322 total pagecache pages 2012-11-19 10:32:56 0 pages in swap cache 2012-11-19 10:32:56 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:32:56 Free swap = 4194176kB 2012-11-19 10:32:56 Total swap = 4194176kB 2012-11-19 10:32:56 1011712 pages RAM 2012-11-19 10:32:56 20742 pages reserved 2012-11-19 10:32:56 20355 pages shared 2012-11-19 10:32:56 980196 pages non-shared 2012-11-19 10:32:56 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:32:56 Call Trace: 2012-11-19 10:32:56 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:32:56 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:32:56 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:32:56 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:32:56 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:32:56 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:32:56 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:32:56 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:32:56 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:32:56 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:32:56 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:32:56 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:32:56 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:32:56 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:32:56 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:32:56 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:32:56 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:32:56 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:32:56 --- Exception: 501 at .pseries_dedicated_idle_sleep+0x11c/0x1f0 2012-11-19 10:32:56 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 10:32:56 [c000000f59b7bd80] [0001d818a21bbf80] 0x1d818a21bbf80 (unreliable) 2012-11-19 10:32:56 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 10:32:56 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:32:56 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:32:56 Mem-Info: 2012-11-19 10:32:56 Node 0 DMA per-cpu: 2012-11-19 10:32:56 CPU 0: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 4: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 6: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 9: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 13: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 14: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 24: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 25: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 26: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 30: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 35: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:56 CPU 36: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 37: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:56 CPU 39: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:56 CPU 40: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:56 CPU 42: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:56 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 44: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 45: hi: 6, btch: 1 usd: 3 2012-11-19 10:32:57 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 active_anon:12234 inactive_anon:5032 isolated_anon:0 2012-11-19 10:32:57 active_file:70682 inactive_file:847955 isolated_file:0 2012-11-19 10:32:57 unevictable:0 dirty:558 writeback:0 unstable:0 2012-11-19 10:32:57 free:633 slab_reclaimable:16989 slab_unreclaimable:18993 2012-11-19 10:32:57 mapped:1641 shmem:78 pagetables:651 bounce:0 2012-11-19 10:32:57 Node 0 DMA free:40512kB min:32128kB low:40128kB high:48192kB active_anon:782976kB inactive_anon:322048kB active_file:4523648kB inactive_file:54269120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:35712kB writeback:0kB mapped:105024kB shmem:4992kB slab_reclaimable:1087296kB slab_unreclaimable:1215552kB kernel_stack:27424kB pagetables:41664kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:32:57 lowmem_reserve[]: 0 0 0 2012-11-19 10:32:57 Node 0 DMA: 516*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 33024kB 2012-11-19 10:32:57 918004 total pagecache pages 2012-11-19 10:32:57 0 pages in swap cache 2012-11-19 10:32:57 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:32:57 Free swap = 4194176kB 2012-11-19 10:32:57 Total swap = 4194176kB 2012-11-19 10:32:57 1011712 pages RAM 2012-11-19 10:32:57 20742 pages reserved 2012-11-19 10:32:57 20360 pages shared 2012-11-19 10:32:57 980189 pages non-shared 2012-11-19 10:32:57 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:32:57 Call Trace: 2012-11-19 10:32:57 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:32:57 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:32:57 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:32:57 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:32:57 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:32:57 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:32:57 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:32:57 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:32:57 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:32:57 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:32:57 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:32:57 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:32:57 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:32:57 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:32:57 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:32:57 --- Exception: 501 at .pseries_dedicated_idle_sleep+0x11c/0x1f0 2012-11-19 10:32:57 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 10:32:57 [c000000f59b7bd80] [0001d818a21bbf80] 0x1d818a21bbf80 (unreliable) 2012-11-19 10:32:57 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 10:32:57 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:32:57 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:32:57 Mem-Info: 2012-11-19 10:32:57 Node 0 DMA per-cpu: 2012-11-19 10:32:57 CPU 0: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 4: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 6: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 9: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 13: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 14: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 24: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 25: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 26: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 30: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 35: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 36: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 37: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 39: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 42: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 44: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 45: hi: 6, btch: 1 usd: 3 2012-11-19 10:32:57 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 active_anon:13408 inactive_anon:5032 isolated_anon:0 2012-11-19 10:32:57 active_file:70682 inactive_file:846771 isolated_file:0 2012-11-19 10:32:57 unevictable:0 dirty:556 writeback:0 unstable:0 2012-11-19 10:32:57 free:642 slab_reclaimable:16989 slab_unreclaimable:18993 2012-11-19 10:32:57 mapped:1641 shmem:78 pagetables:651 bounce:0 2012-11-19 10:32:57 Node 0 DMA free:41088kB min:32128kB low:40128kB high:48192kB active_anon:858112kB inactive_anon:322048kB active_file:4523648kB inactive_file:54193344kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:35584kB writeback:0kB mapped:105024kB shmem:4992kB slab_reclaimable:1087296kB slab_unreclaimable:1215552kB kernel_stack:27424kB pagetables:41664kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:32:57 lowmem_reserve[]: 0 0 0 2012-11-19 10:32:57 Node 0 DMA: 518*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 33280kB 2012-11-19 10:32:57 916980 total pagecache pages 2012-11-19 10:32:57 0 pages in swap cache 2012-11-19 10:32:57 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:32:57 Free swap = 4194176kB 2012-11-19 10:32:57 Total swap = 4194176kB 2012-11-19 10:32:57 1011712 pages RAM 2012-11-19 10:32:57 20742 pages reserved 2012-11-19 10:32:57 20361 pages shared 2012-11-19 10:32:57 980183 pages non-shared 2012-11-19 10:32:57 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:32:57 Call Trace: 2012-11-19 10:32:57 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:32:57 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:32:57 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:32:57 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:32:57 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:32:57 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:32:57 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:32:57 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:32:57 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:32:57 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:32:57 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:32:57 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:32:57 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:32:57 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:32:57 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:32:57 --- Exception: 501 at .pseries_dedicated_idle_sleep+0x11c/0x1f0 2012-11-19 10:32:57 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 10:32:57 [c000000f59b7bd80] [0001d818a21bbf80] 0x1d818a21bbf80 (unreliable) 2012-11-19 10:32:57 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 10:32:57 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:32:57 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:32:57 Mem-Info: 2012-11-19 10:32:57 Node 0 DMA per-cpu: 2012-11-19 10:32:57 CPU 0: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 4: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 6: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 9: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 13: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 14: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 24: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 25: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 26: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 30: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 35: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 36: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 37: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 39: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 42: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 44: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 45: hi: 6, btch: 1 usd: 3 2012-11-19 10:32:57 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 active_anon:14389 inactive_anon:5032 isolated_anon:0 2012-11-19 10:32:57 active_file:70681 inactive_file:845779 isolated_file:0 2012-11-19 10:32:57 unevictable:0 dirty:490 writeback:0 unstable:0 2012-11-19 10:32:57 free:642 slab_reclaimable:16989 slab_unreclaimable:18992 2012-11-19 10:32:57 mapped:1641 shmem:78 pagetables:651 bounce:0 2012-11-19 10:32:57 Node 0 DMA free:41088kB min:32128kB low:40128kB high:48192kB active_anon:920896kB inactive_anon:322048kB active_file:4523584kB inactive_file:54129856kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:31360kB writeback:0kB mapped:105024kB shmem:4992kB slab_reclaimable:1087296kB slab_unreclaimable:1215488kB kernel_stack:27424kB pagetables:41664kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:32:57 lowmem_reserve[]: 0 0 0 2012-11-19 10:32:57 Node 0 DMA: 527*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 33728kB 2012-11-19 10:32:57 915763 total pagecache pages 2012-11-19 10:32:57 0 pages in swap cache 2012-11-19 10:32:57 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:32:57 Free swap = 4194176kB 2012-11-19 10:32:57 Total swap = 4194176kB 2012-11-19 10:32:57 1011712 pages RAM 2012-11-19 10:32:57 20742 pages reserved 2012-11-19 10:32:57 20362 pages shared 2012-11-19 10:32:57 980190 pages non-shared 2012-11-19 10:32:57 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:32:57 Call Trace: 2012-11-19 10:32:57 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:32:57 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:32:57 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:32:57 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:32:57 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:32:57 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:32:57 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:32:57 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:32:57 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:32:57 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:32:57 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:32:57 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:32:57 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:32:57 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:32:57 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:32:57 --- Exception: 501 at .pseries_dedicated_idle_sleep+0x11c/0x1f0 2012-11-19 10:32:57 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 10:32:57 [c000000f59b7bd80] [0001d818a21bbf80] 0x1d818a21bbf80 (unreliable) 2012-11-19 10:32:57 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 10:32:57 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:32:57 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:32:57 Mem-Info: 2012-11-19 10:32:57 Node 0 DMA per-cpu: 2012-11-19 10:32:57 CPU 0: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 4: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 6: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 9: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 13: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 14: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 24: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 25: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 26: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 30: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 35: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 36: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 37: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 39: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 42: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 44: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 45: hi: 6, btch: 1 usd: 3 2012-11-19 10:32:57 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 active_anon:15586 inactive_anon:5032 isolated_anon:0 2012-11-19 10:32:57 active_file:70681 inactive_file:844595 isolated_file:0 2012-11-19 10:32:57 unevictable:0 dirty:455 writeback:0 unstable:0 2012-11-19 10:32:57 free:634 slab_reclaimable:16989 slab_unreclaimable:18995 2012-11-19 10:32:57 mapped:1641 shmem:78 pagetables:651 bounce:0 2012-11-19 10:32:57 Node 0 DMA free:40576kB min:32128kB low:40128kB high:48192kB active_anon:997504kB inactive_anon:322048kB active_file:4523584kB inactive_file:54054080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:29120kB writeback:0kB mapped:105024kB shmem:4992kB slab_reclaimable:1087296kB slab_unreclaimable:1215680kB kernel_stack:27424kB pagetables:41664kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:32:57 lowmem_reserve[]: 0 0 0 2012-11-19 10:32:57 Node 0 DMA: 512*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 32896kB 2012-11-19 10:32:57 914899 total pagecache pages 2012-11-19 10:32:57 0 pages in swap cache 2012-11-19 10:32:57 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:32:57 Free swap = 4194176kB 2012-11-19 10:32:57 Total swap = 4194176kB 2012-11-19 10:32:57 1011712 pages RAM 2012-11-19 10:32:57 20742 pages reserved 2012-11-19 10:32:57 20362 pages shared 2012-11-19 10:32:57 980181 pages non-shared 2012-11-19 10:32:57 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:32:57 Call Trace: 2012-11-19 10:32:57 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:32:57 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:32:57 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:32:57 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:32:57 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:32:57 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:32:57 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:32:57 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:32:57 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:32:57 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:32:57 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:32:57 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:32:57 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:32:57 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:32:57 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:32:57 --- Exception: 501 at .pseries_dedicated_idle_sleep+0x11c/0x1f0 2012-11-19 10:32:57 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 10:32:57 [c000000f59b7bd80] [0001d818a21bbf80] 0x1d818a21bbf80 (unreliable) 2012-11-19 10:32:57 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 10:32:57 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:32:57 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:32:57 Mem-Info: 2012-11-19 10:32:57 Node 0 DMA per-cpu: 2012-11-19 10:32:57 CPU 0: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 4: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 6: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 9: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 13: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 14: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 24: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 25: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 26: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 30: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 35: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 36: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 37: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 39: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 42: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 44: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 45: hi: 6, btch: 1 usd: 3 2012-11-19 10:32:57 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 active_anon:16541 inactive_anon:5032 isolated_anon:0 2012-11-19 10:32:57 active_file:70681 inactive_file:843667 isolated_file:0 2012-11-19 10:32:57 unevictable:0 dirty:419 writeback:0 unstable:0 2012-11-19 10:32:57 free:597 slab_reclaimable:16985 slab_unreclaimable:18999 2012-11-19 10:32:57 mapped:1641 shmem:78 pagetables:650 bounce:0 2012-11-19 10:32:57 Node 0 DMA free:41088kB min:32128kB low:40128kB high:48192kB active_anon:1064000kB inactive_anon:322048kB active_file:4523584kB inactive_file:53986496kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:26816kB writeback:0kB mapped:105024kB shmem:4992kB slab_reclaimable:1086848kB slab_unreclaimable:1215936kB kernel_stack:27424kB pagetables:41600kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:32:57 lowmem_reserve[]: 0 0 0 2012-11-19 10:32:57 Node 0 DMA: 577*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 36928kB 2012-11-19 10:32:57 913843 total pagecache pages 2012-11-19 10:32:57 0 pages in swap cache 2012-11-19 10:32:57 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:32:57 Free swap = 4194176kB 2012-11-19 10:32:57 Total swap = 4194176kB 2012-11-19 10:32:57 1011712 pages RAM 2012-11-19 10:32:57 20742 pages reserved 2012-11-19 10:32:57 20368 pages shared 2012-11-19 10:32:57 980119 pages non-shared 2012-11-19 10:32:57 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:32:57 Call Trace: 2012-11-19 10:32:57 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:32:57 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:32:57 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:32:57 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:32:57 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:32:57 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:32:57 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:32:57 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:32:57 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:32:57 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:32:57 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:32:57 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:32:57 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:32:57 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:32:57 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:32:57 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:32:57 --- Exception: 501 at .pseries_dedicated_idle_sleep+0x11c/0x1f0 2012-11-19 10:32:57 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 10:32:57 [c000000f59b7bd80] [0001d818a21bbf80] 0x1d818a21bbf80 (unreliable) 2012-11-19 10:32:57 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 10:32:57 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:32:57 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:32:57 Mem-Info: 2012-11-19 10:32:57 Node 0 DMA per-cpu: 2012-11-19 10:32:57 CPU 0: hi: 6, btch: 1 usd: 3 2012-11-19 10:32:57 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 4: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 6: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 9: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 13: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 14: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 24: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 25: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 26: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 30: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 35: hi: 6, btch: 1 usd: 4 2012-11-19 10:32:57 CPU 36: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 37: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 39: hi: 6, btch: 1 usd: 2 2012-11-19 10:32:57 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 10:32:57 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 42: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 44: hi: 6, btch: 1 usd: 1 2012-11-19 10:32:57 CPU 45: hi: 6, btch: 1 usd: 3 2012-11-19 10:32:57 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 10:32:57 active_anon:17556 inactive_anon:5032 isolated_anon:0 2012-11-19 10:32:57 active_file:70681 inactive_file:842595 isolated_file:0 2012-11-19 10:32:57 unevictable:0 dirty:198 writeback:1 unstable:0 2012-11-19 10:32:57 free:652 slab_reclaimable:16980 slab_unreclaimable:18999 2012-11-19 10:32:57 mapped:1641 shmem:78 pagetables:649 bounce:0 2012-11-19 10:32:57 Node 0 DMA free:41728kB min:32128kB low:40128kB high:48192kB active_anon:1123584kB inactive_anon:322048kB active_file:4523584kB inactive_file:53926080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:12672kB writeback:64kB mapped:105024kB shmem:4992kB slab_reclaimable:1086720kB slab_unreclaimable:1215936kB kernel_stack:27248kB pagetables:41536kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:32:57 lowmem_reserve[]: 0 0 0 2012-11-19 10:32:57 Node 0 DMA: 464*64kB 20*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 32256kB 2012-11-19 10:32:57 912630 total pagecache pages 2012-11-19 10:32:57 0 pages in swap cache 2012-11-19 10:32:57 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:32:57 Free swap = 4194176kB 2012-11-19 10:32:57 Total swap = 4194176kB 2012-11-19 10:32:57 1011712 pages RAM 2012-11-19 10:32:57 20742 pages reserved 2012-11-19 10:32:57 20359 pages shared 2012-11-19 10:32:57 980171 pages non-shared 2012-11-19 10:38:34 rpciod/8: page allocation failure. order:1, mode:0x20 2012-11-19 10:38:34 Call Trace: 2012-11-19 10:38:34 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:38:34 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:38:34 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:38:34 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:38:34 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:38:34 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:38:34 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:38:34 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:38:34 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:38:34 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:38:34 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:38:34 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:38:34 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:38:34 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:38:34 [c000000f498c6920] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:38:34 [c000000f498c69c0] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:38:34 [c000000f498c6a40] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:38:34 [c000000f498c6af0] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:38:34 --- Exception: 501 at .memcpy+0x60/0x280 2012-11-19 10:38:34 LR = .pskb_expand_head+0xb0/0x280 2012-11-19 10:38:34 [c000000f498c6de0] [c0000000004d6b00] .pskb_expand_head+0x70/0x280 (unreliable) 2012-11-19 10:38:34 [c000000f498c6e90] [c0000000004d7358] .__pskb_pull_tail+0x148/0x420 2012-11-19 10:38:34 [c000000f498c6f40] [c0000000004e6e3c] .dev_queue_xmit+0x62c/0x8e0 2012-11-19 10:38:34 [c000000f498c6ff0] [c0000000004efc70] .neigh_connected_output+0x110/0x160 2012-11-19 10:38:34 [c000000f498c70a0] [c00000000053079c] .ip_finish_output+0x6c/0x3e0 2012-11-19 10:38:34 [c000000f498c7140] [c000000000530fa4] .ip_local_out+0x44/0x60 2012-11-19 10:38:34 [c000000f498c71c0] [c0000000005314fc] .ip_queue_xmit+0x16c/0x410 2012-11-19 10:38:34 [c000000f498c72c0] [c000000000549fd4] .tcp_transmit_skb+0x474/0x9b0 2012-11-19 10:38:34 [c000000f498c73c0] [c00000000054d020] .tcp_write_xmit+0x230/0xbb0 2012-11-19 10:38:34 [c000000f498c74d0] [c00000000054da20] .__tcp_push_pending_frames+0x40/0x100 2012-11-19 10:38:34 [c000000f498c7550] [c00000000053d228] .tcp_sendmsg+0x2a8/0xdf0 2012-11-19 10:38:34 [c000000f498c76a0] [c0000000004ca3ec] .sock_sendmsg+0x14c/0x1a0 2012-11-19 10:38:34 [c000000f498c78c0] [c0000000004ca478] .kernel_sendmsg+0x38/0x60 2012-11-19 10:38:34 [c000000f498c7940] [d00000000bfe8ac4] .xs_send_kvec+0xb4/0xd0 [sunrpc] 2012-11-19 10:38:34 [c000000f498c7a00] [d00000000bfe8b9c] .xs_sendpages+0xbc/0x350 [sunrpc] 2012-11-19 10:38:34 [c000000f498c7ad0] [d00000000bfe8fdc] .xs_tcp_send_request+0x6c/0x1e0 [sunrpc] 2012-11-19 10:38:34 [c000000f498c7b80] [d00000000bfe59a4] .xprt_transmit+0xa4/0x340 [sunrpc] 2012-11-19 10:38:34 [c000000f498c7c40] [d00000000bfe1928] .call_transmit+0x1e8/0x2c0 [sunrpc] 2012-11-19 10:38:34 [c000000f498c7cd0] [d00000000bfef154] .__rpc_execute+0xc4/0x470 [sunrpc] 2012-11-19 10:38:34 [c000000f498c7db0] [c0000000000b6cac] .worker_thread+0x1dc/0x3c0 2012-11-19 10:38:34 [c000000f498c7ed0] [c0000000000bd02c] .kthread+0xbc/0xd0 2012-11-19 10:38:34 [c000000f498c7f90] [c000000000032054] .kernel_thread+0x54/0x70 2012-11-19 10:38:34 Mem-Info: 2012-11-19 10:38:34 Node 0 DMA per-cpu: 2012-11-19 10:38:34 CPU 0: hi: 6, btch: 1 usd: 3 2012-11-19 10:38:34 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 5: hi: 6, btch: 1 usd: 4 2012-11-19 10:38:34 CPU 6: hi: 6, btch: 1 usd: 4 2012-11-19 10:38:34 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 9: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 10: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 13: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 16: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 24: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 25: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 26: hi: 6, btch: 1 usd: 4 2012-11-19 10:38:34 CPU 27: hi: 6, btch: 1 usd: 3 2012-11-19 10:38:34 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 29: hi: 6, btch: 1 usd: 3 2012-11-19 10:38:34 CPU 30: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 33: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 35: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 36: hi: 6, btch: 1 usd: 4 2012-11-19 10:38:34 CPU 37: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 40: hi: 6, btch: 1 usd: 1 2012-11-19 10:38:34 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 42: hi: 6, btch: 1 usd: 1 2012-11-19 10:38:34 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 44: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 CPU 45: hi: 6, btch: 1 usd: 1 2012-11-19 10:38:34 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 10:38:34 CPU 47: hi: 6, btch: 1 usd: 0 2012-11-19 10:38:34 active_anon:55823 inactive_anon:5032 isolated_anon:0 2012-11-19 10:38:34 active_file:68335 inactive_file:809491 isolated_file:0 2012-11-19 10:38:34 unevictable:0 dirty:56 writeback:112 unstable:0 2012-11-19 10:38:34 free:1109 slab_reclaimable:14699 slab_unreclaimable:18098 2012-11-19 10:38:34 mapped:1656 shmem:78 pagetables:599 bounce:0 2012-11-19 10:38:34 Node 0 DMA free:70976kB min:32128kB low:40128kB high:48192kB active_anon:3572672kB inactive_anon:322048kB active_file:4373440kB inactive_file:51807424kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:3584kB writeback:7168kB mapped:105984kB shmem:4992kB slab_reclaimable:940736kB slab_unreclaimable:1158272kB kernel_stack:26816kB pagetables:38336kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:38:34 lowmem_reserve[]: 0 0 0 2012-11-19 10:38:34 Node 0 DMA: 649*64kB 103*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 54720kB 2012-11-19 10:38:34 877967 total pagecache pages 2012-11-19 10:38:34 0 pages in swap cache 2012-11-19 10:38:34 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:38:34 Free swap = 4194176kB 2012-11-19 10:38:34 Total swap = 4194176kB 2012-11-19 10:38:34 1011712 pages RAM 2012-11-19 10:38:34 20742 pages reserved 2012-11-19 10:38:34 21409 pages shared 2012-11-19 10:38:34 977738 pages non-shared 2012-11-19 10:40:15 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 10:40:15 Call Trace: 2012-11-19 10:40:15 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:40:15 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:40:15 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:40:15 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:40:15 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:40:15 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:40:15 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:40:15 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:40:15 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:40:15 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:40:15 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:40:15 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:40:15 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:40:15 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:40:15 [c000000f59b7b970] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:40:15 [c000000f59b7ba10] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:40:15 [c000000f59b7ba90] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:40:15 [c000000f59b7bb40] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:40:15 --- Exception: 501 at .cpu_idle+0x15c/0x1d0 2012-11-19 10:40:15 LR = .cpu_idle+0x14c/0x1d0 2012-11-19 10:40:15 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 (unreliable) 2012-11-19 10:40:15 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 10:40:15 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 10:40:15 Mem-Info: 2012-11-19 10:40:15 Node 0 DMA per-cpu: 2012-11-19 10:40:15 CPU 0: hi: 6, btch: 1 usd: 2 2012-11-19 10:40:15 CPU 1: hi: 6, btch: 1 usd: 1 2012-11-19 10:40:15 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 9: hi: 6, btch: 1 usd: 3 2012-11-19 10:40:15 CPU 10: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 11: hi: 6, btch: 1 usd: 4 2012-11-19 10:40:15 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 13: hi: 6, btch: 1 usd: 2 2012-11-19 10:40:15 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 17: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 21: hi: 6, btch: 1 usd: 3 2012-11-19 10:40:15 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 24: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 CPU 25: hi: 6, btch: 1 usd: 1 2012-11-19 10:40:15 CPU 26: hi: 6, btch: 1 usd: 2 2012-11-19 10:40:15 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 CPU 28: hi: 6, btch: 1 usd: 1 2012-11-19 10:40:15 CPU 29: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 30: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 CPU 33: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 CPU 35: hi: 6, btch: 1 usd: 2 2012-11-19 10:40:15 CPU 36: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 CPU 37: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 CPU 38: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 CPU 40: hi: 6, btch: 1 usd: 2 2012-11-19 10:40:15 CPU 41: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 CPU 42: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 10:40:15 CPU 44: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 CPU 45: hi: 6, btch: 1 usd: 2 2012-11-19 10:40:15 CPU 46: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 CPU 47: hi: 6, btch: 1 usd: 0 2012-11-19 10:40:15 active_anon:95982 inactive_anon:5032 isolated_anon:0 2012-11-19 10:40:15 active_file:68302 inactive_file:770499 isolated_file:0 2012-11-19 10:40:15 unevictable:0 dirty:690 writeback:0 unstable:0 2012-11-19 10:40:15 free:891 slab_reclaimable:13928 slab_unreclaimable:18054 2012-11-19 10:40:15 mapped:1701 shmem:78 pagetables:609 bounce:0 2012-11-19 10:40:15 Node 0 DMA free:57024kB min:32128kB low:40128kB high:48192kB active_anon:6142848kB inactive_anon:322048kB active_file:4371328kB inactive_file:49311936kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:44160kB writeback:0kB mapped:108864kB shmem:4992kB slab_reclaimable:891392kB slab_unreclaimable:1155456kB kernel_stack:26816kB pagetables:38976kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:40:15 lowmem_reserve[]: 0 0 0 2012-11-19 10:40:15 Node 0 DMA: 271*64kB 95*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 29504kB 2012-11-19 10:40:15 839048 total pagecache pages 2012-11-19 10:40:15 0 pages in swap cache 2012-11-19 10:40:15 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:40:15 Free swap = 4194176kB 2012-11-19 10:40:15 Total swap = 4194176kB 2012-11-19 10:40:15 1011712 pages RAM 2012-11-19 10:40:15 20742 pages reserved 2012-11-19 10:40:15 21438 pages shared 2012-11-19 10:40:15 978144 pages non-shared 2012-11-19 10:56:09 rpciod/8: page allocation failure. order:1, mode:0x20 2012-11-19 10:56:09 Call Trace: 2012-11-19 10:56:09 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:56:09 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:56:09 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:56:09 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:56:09 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:56:09 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:56:09 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:56:09 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:56:09 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:56:09 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:56:09 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:56:09 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:56:09 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:56:09 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:56:09 [c000000f498c6c00] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:56:09 [c000000f498c6ca0] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 10:56:09 [c000000f498c6d20] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 10:56:09 [c000000f498c6dd0] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 10:56:09 --- Exception: 501 at .ip_output+0x70/0x110 2012-11-19 10:56:09 LR = .ip_local_out+0x44/0x60 2012-11-19 10:56:09 [c000000f498c70c0] [c000000000530bfc] .ip_output+0xec/0x110 (unreliable) 2012-11-19 10:56:09 [c000000f498c7140] [c000000000530fa4] .ip_local_out+0x44/0x60 2012-11-19 10:56:09 [c000000f498c71c0] [c0000000005314fc] .ip_queue_xmit+0x16c/0x410 2012-11-19 10:56:09 [c000000f498c72c0] [c000000000549fd4] .tcp_transmit_skb+0x474/0x9b0 2012-11-19 10:56:09 [c000000f498c73c0] [c00000000054d020] .tcp_write_xmit+0x230/0xbb0 2012-11-19 10:56:09 [c000000f498c74d0] [c00000000054da20] .__tcp_push_pending_frames+0x40/0x100 2012-11-19 10:56:09 [c000000f498c7550] [c00000000053d228] .tcp_sendmsg+0x2a8/0xdf0 2012-11-19 10:56:09 [c000000f498c76a0] [c0000000004ca3ec] .sock_sendmsg+0x14c/0x1a0 2012-11-19 10:56:09 [c000000f498c78c0] [c0000000004ca478] .kernel_sendmsg+0x38/0x60 2012-11-19 10:56:09 [c000000f498c7940] [d00000000bfe8ac4] .xs_send_kvec+0xb4/0xd0 [sunrpc] 2012-11-19 10:56:09 [c000000f498c7a00] [d00000000bfe8b9c] .xs_sendpages+0xbc/0x350 [sunrpc] 2012-11-19 10:56:09 [c000000f498c7ad0] [d00000000bfe8fdc] .xs_tcp_send_request+0x6c/0x1e0 [sunrpc] 2012-11-19 10:56:09 [c000000f498c7b80] [d00000000bfe59a4] .xprt_transmit+0xa4/0x340 [sunrpc] 2012-11-19 10:56:09 [c000000f498c7c40] [d00000000bfe1928] .call_transmit+0x1e8/0x2c0 [sunrpc] 2012-11-19 10:56:09 [c000000f498c7cd0] [d00000000bfef154] .__rpc_execute+0xc4/0x470 [sunrpc] 2012-11-19 10:56:09 [c000000f498c7db0] [c0000000000b6cac] .worker_thread+0x1dc/0x3c0 2012-11-19 10:56:09 [c000000f498c7ed0] [c0000000000bd02c] .kthread+0xbc/0xd0 2012-11-19 10:56:09 [c000000f498c7f90] [c000000000032054] .kernel_thread+0x54/0x70 2012-11-19 10:56:09 Mem-Info: 2012-11-19 10:56:09 Node 0 DMA per-cpu: 2012-11-19 10:56:09 CPU 0: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 2: hi: 6, btch: 1 usd: 3 2012-11-19 10:56:09 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 5: hi: 6, btch: 1 usd: 2 2012-11-19 10:56:09 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 9: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 10: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 13: hi: 6, btch: 1 usd: 2 2012-11-19 10:56:09 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 16: hi: 6, btch: 1 usd: 1 2012-11-19 10:56:09 CPU 17: hi: 6, btch: 1 usd: 2 2012-11-19 10:56:09 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 21: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 24: hi: 6, btch: 1 usd: 1 2012-11-19 10:56:09 CPU 25: hi: 6, btch: 1 usd: 2 2012-11-19 10:56:09 CPU 26: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 28: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 29: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 30: hi: 6, btch: 1 usd: 1 2012-11-19 10:56:09 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 32: hi: 6, btch: 1 usd: 3 2012-11-19 10:56:09 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 10:56:09 CPU 34: hi: 6, btch: 1 usd: 5 2012-11-19 10:56:09 CPU 35: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 36: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 37: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 41: hi: 6, btch: 1 usd: 2 2012-11-19 10:56:09 CPU 42: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 44: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 45: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 46: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 CPU 47: hi: 6, btch: 1 usd: 0 2012-11-19 10:56:09 active_anon:59086 inactive_anon:5032 isolated_anon:0 2012-11-19 10:56:09 active_file:67893 inactive_file:810809 isolated_file:0 2012-11-19 10:56:09 unevictable:0 dirty:73 writeback:0 unstable:0 2012-11-19 10:56:09 free:925 slab_reclaimable:12086 slab_unreclaimable:16733 2012-11-19 10:56:09 mapped:1656 shmem:78 pagetables:600 bounce:0 2012-11-19 10:56:09 Node 0 DMA free:59200kB min:32128kB low:40128kB high:48192kB active_anon:3781504kB inactive_anon:322048kB active_file:4345152kB inactive_file:51891776kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:4672kB writeback:0kB mapped:105984kB shmem:4992kB slab_reclaimable:773504kB slab_unreclaimable:1070912kB kernel_stack:26800kB pagetables:38400kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:56:09 lowmem_reserve[]: 0 0 0 2012-11-19 10:56:09 Node 0 DMA: 416*64kB 103*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 39808kB 2012-11-19 10:56:09 878367 total pagecache pages 2012-11-19 10:56:09 0 pages in swap cache 2012-11-19 10:56:09 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:56:09 Free swap = 4194176kB 2012-11-19 10:56:09 Total swap = 4194176kB 2012-11-19 10:56:09 1011712 pages RAM 2012-11-19 10:56:09 20742 pages reserved 2012-11-19 10:56:09 22221 pages shared 2012-11-19 10:56:09 977053 pages non-shared 2012-11-19 10:57:44 rpciod/8: page allocation failure. order:1, mode:0x20 2012-11-19 10:57:44 Call Trace: 2012-11-19 10:57:44 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 10:57:44 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 10:57:44 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 10:57:44 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 10:57:44 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 10:57:44 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 10:57:44 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 10:57:44 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 10:57:44 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 10:57:44 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 10:57:44 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 10:57:44 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 10:57:44 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 10:57:44 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 10:57:44 [c000000f498c6e20] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 10:57:44 [c000000f498c6ec0] [c00000000009c92c] .local_bh_enable+0xbc/0xd0 2012-11-19 10:57:44 [c000000f498c6f40] [c0000000004e6a6c] .dev_queue_xmit+0x25c/0x8e0 2012-11-19 10:57:44 [c000000f498c6ff0] [c0000000004efc70] .neigh_connected_output+0x110/0x160 2012-11-19 10:57:44 [c000000f498c70a0] [c00000000053079c] .ip_finish_output+0x6c/0x3e0 2012-11-19 10:57:44 [c000000f498c7140] [c000000000530fa4] .ip_local_out+0x44/0x60 2012-11-19 10:57:44 [c000000f498c71c0] [c0000000005314fc] .ip_queue_xmit+0x16c/0x410 2012-11-19 10:57:44 [c000000f498c72c0] [c000000000549fd4] .tcp_transmit_skb+0x474/0x9b0 2012-11-19 10:57:44 [c000000f498c73c0] [c00000000054d020] .tcp_write_xmit+0x230/0xbb0 2012-11-19 10:57:44 [c000000f498c74d0] [c00000000054da20] .__tcp_push_pending_frames+0x40/0x100 2012-11-19 10:57:44 [c000000f498c7550] [c00000000053d228] .tcp_sendmsg+0x2a8/0xdf0 2012-11-19 10:57:44 [c000000f498c76a0] [c0000000004ca3ec] .sock_sendmsg+0x14c/0x1a0 2012-11-19 10:57:44 [c000000f498c78c0] [c0000000004ca478] .kernel_sendmsg+0x38/0x60 2012-11-19 10:57:44 [c000000f498c7940] [d00000000bfe8ac4] .xs_send_kvec+0xb4/0xd0 [sunrpc] 2012-11-19 10:57:44 [c000000f498c7a00] [d00000000bfe8b9c] .xs_sendpages+0xbc/0x350 [sunrpc] 2012-11-19 10:57:44 [c000000f498c7ad0] [d00000000bfe8fdc] .xs_tcp_send_request+0x6c/0x1e0 [sunrpc] 2012-11-19 10:57:44 [c000000f498c7b80] [d00000000bfe59a4] .xprt_transmit+0xa4/0x340 [sunrpc] 2012-11-19 10:57:44 [c000000f498c7c40] [d00000000bfe1928] .call_transmit+0x1e8/0x2c0 [sunrpc] 2012-11-19 10:57:44 [c000000f498c7cd0] [d00000000bfef154] .__rpc_execute+0xc4/0x470 [sunrpc] 2012-11-19 10:57:44 [c000000f498c7db0] [c0000000000b6cac] .worker_thread+0x1dc/0x3c0 2012-11-19 10:57:44 [c000000f498c7ed0] [c0000000000bd02c] .kthread+0xbc/0xd0 2012-11-19 10:57:44 [c000000f498c7f90] [c000000000032054] .kernel_thread+0x54/0x70 2012-11-19 10:57:44 Mem-Info: 2012-11-19 10:57:44 Node 0 DMA per-cpu: 2012-11-19 10:57:44 CPU 0: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 CPU 1: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 5: hi: 6, btch: 1 usd: 3 2012-11-19 10:57:44 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 9: hi: 6, btch: 1 usd: 6 2012-11-19 10:57:44 CPU 10: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 17: hi: 6, btch: 1 usd: 3 2012-11-19 10:57:44 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 20: hi: 6, btch: 1 usd: 4 2012-11-19 10:57:44 CPU 21: hi: 6, btch: 1 usd: 3 2012-11-19 10:57:44 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 24: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 CPU 25: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 26: hi: 6, btch: 1 usd: 4 2012-11-19 10:57:44 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 CPU 29: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 CPU 30: hi: 6, btch: 1 usd: 2 2012-11-19 10:57:44 CPU 31: hi: 6, btch: 1 usd: 4 2012-11-19 10:57:44 CPU 32: hi: 6, btch: 1 usd: 2 2012-11-19 10:57:44 CPU 33: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 CPU 34: hi: 6, btch: 1 usd: 4 2012-11-19 10:57:44 CPU 35: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 36: hi: 6, btch: 1 usd: 3 2012-11-19 10:57:44 CPU 37: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 CPU 41: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 CPU 42: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 CPU 44: hi: 6, btch: 1 usd: 5 2012-11-19 10:57:44 CPU 45: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 CPU 46: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 CPU 47: hi: 6, btch: 1 usd: 0 2012-11-19 10:57:44 active_anon:100325 inactive_anon:5032 isolated_anon:0 2012-11-19 10:57:44 active_file:66270 inactive_file:772248 isolated_file:0 2012-11-19 10:57:44 unevictable:0 dirty:353 writeback:0 unstable:0 2012-11-19 10:57:44 free:980 slab_reclaimable:11175 slab_unreclaimable:16559 2012-11-19 10:57:44 mapped:1701 shmem:78 pagetables:610 bounce:0 2012-11-19 10:57:44 Node 0 DMA free:62720kB min:32128kB low:40128kB high:48192kB active_anon:6420800kB inactive_anon:322048kB active_file:4241280kB inactive_file:49423872kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:22592kB writeback:0kB mapped:108864kB shmem:4992kB slab_reclaimable:715200kB slab_unreclaimable:1059776kB kernel_stack:26800kB pagetables:39040kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 10:57:44 lowmem_reserve[]: 0 0 0 2012-11-19 10:57:44 Node 0 DMA: 768*64kB 101*128kB 3*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 62848kB 2012-11-19 10:57:44 838231 total pagecache pages 2012-11-19 10:57:44 0 pages in swap cache 2012-11-19 10:57:44 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 10:57:44 Free swap = 4194176kB 2012-11-19 10:57:44 Total swap = 4194176kB 2012-11-19 10:57:44 1011712 pages RAM 2012-11-19 10:57:44 20742 pages reserved 2012-11-19 10:57:44 30337 pages shared 2012-11-19 10:57:44 970630 pages non-shared Console [seqlac2] log at 2012-11-19 11:00:00 PST. 2012-11-19 11:01:18 rpciod/8: page allocation failure. order:1, mode:0x20 2012-11-19 11:01:18 Call Trace: 2012-11-19 11:01:18 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 11:01:18 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 11:01:18 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 11:01:18 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 11:01:18 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 11:01:18 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 11:01:18 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 11:01:18 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 11:01:18 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 11:01:18 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 11:01:18 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 11:01:18 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 11:01:18 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 11:01:18 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 11:01:18 [c000000f498c6be0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 11:01:18 [c000000f498c6c80] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 11:01:18 [c000000f498c6d00] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 11:01:18 [c000000f498c6db0] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 11:01:18 --- Exception: 501 at .neigh_connected_output+0x10/0x160 2012-11-19 11:01:18 LR = .ip_finish_output+0x6c/0x3e0 2012-11-19 11:01:18 [c000000f498c70a0] [c000000000ed3e48] mpic_set_clk_ratio+0x0/0x10 (unreliable) 2012-11-19 11:01:18 [c000000f498c7140] [c000000000530fa4] .ip_local_out+0x44/0x60 2012-11-19 11:01:18 [c000000f498c71c0] [c0000000005314fc] .ip_queue_xmit+0x16c/0x410 2012-11-19 11:01:18 [c000000f498c72c0] [c000000000549fd4] .tcp_transmit_skb+0x474/0x9b0 2012-11-19 11:01:18 [c000000f498c73c0] [c00000000054d020] .tcp_write_xmit+0x230/0xbb0 2012-11-19 11:01:18 [c000000f498c74d0] [c00000000054da20] .__tcp_push_pending_frames+0x40/0x100 2012-11-19 11:01:18 [c000000f498c7550] [c00000000053d228] .tcp_sendmsg+0x2a8/0xdf0 2012-11-19 11:01:18 [c000000f498c76a0] [c0000000004ca3ec] .sock_sendmsg+0x14c/0x1a0 2012-11-19 11:01:18 [c000000f498c78c0] [c0000000004ca478] .kernel_sendmsg+0x38/0x60 2012-11-19 11:01:18 [c000000f498c7940] [d00000000bfe8ac4] .xs_send_kvec+0xb4/0xd0 [sunrpc] 2012-11-19 11:01:18 [c000000f498c7a00] [d00000000bfe8b9c] .xs_sendpages+0xbc/0x350 [sunrpc] 2012-11-19 11:01:18 [c000000f498c7ad0] [d00000000bfe8fdc] .xs_tcp_send_request+0x6c/0x1e0 [sunrpc] 2012-11-19 11:01:18 [c000000f498c7b80] [d00000000bfe59a4] .xprt_transmit+0xa4/0x340 [sunrpc] 2012-11-19 11:01:18 [c000000f498c7c40] [d00000000bfe1928] .call_transmit+0x1e8/0x2c0 [sunrpc] 2012-11-19 11:01:18 [c000000f498c7cd0] [d00000000bfef154] .__rpc_execute+0xc4/0x470 [sunrpc] 2012-11-19 11:01:18 [c000000f498c7db0] [c0000000000b6cac] .worker_thread+0x1dc/0x3c0 2012-11-19 11:01:18 [c000000f498c7ed0] [c0000000000bd02c] .kthread+0xbc/0xd0 2012-11-19 11:01:18 [c000000f498c7f90] [c000000000032054] .kernel_thread+0x54/0x70 2012-11-19 11:01:18 Mem-Info: 2012-11-19 11:01:18 Node 0 DMA per-cpu: 2012-11-19 11:01:18 CPU 0: hi: 6, btch: 1 usd: 0 2012-11-19 11:01:18 CPU 1: hi: 6, btch: 1 usd: 3 2012-11-19 11:01:18 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 5: hi: 6, btch: 1 usd: 1 2012-11-19 11:01:18 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 8: hi: 6, btch: 1 usd: 0 2012-11-19 11:01:18 CPU 9: hi: 6, btch: 1 usd: 4 2012-11-19 11:01:18 CPU 10: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 13: hi: 6, btch: 1 usd: 0 2012-11-19 11:01:18 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 16: hi: 6, btch: 1 usd: 0 2012-11-19 11:01:18 CPU 17: hi: 6, btch: 1 usd: 2 2012-11-19 11:01:18 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 20: hi: 6, btch: 1 usd: 4 2012-11-19 11:01:18 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 24: hi: 6, btch: 1 usd: 3 2012-11-19 11:01:18 CPU 25: hi: 6, btch: 1 usd: 0 2012-11-19 11:01:18 CPU 26: hi: 6, btch: 1 usd: 3 2012-11-19 11:01:18 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 11:01:18 CPU 28: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 29: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 30: hi: 6, btch: 1 usd: 2 2012-11-19 11:01:18 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 11:01:18 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 33: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 34: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 35: hi: 6, btch: 1 usd: 0 2012-11-19 11:01:18 CPU 36: hi: 6, btch: 1 usd: 0 2012-11-19 11:01:18 CPU 37: hi: 6, btch: 1 usd: 4 2012-11-19 11:01:18 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 11:01:18 CPU 39: hi: 6, btch: 1 usd: 1 2012-11-19 11:01:18 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 11:01:18 CPU 41: hi: 6, btch: 1 usd: 4 2012-11-19 11:01:18 CPU 42: hi: 6, btch: 1 usd: 4 2012-11-19 11:01:18 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 11:01:18 CPU 44: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 11:01:18 CPU 46: hi: 6, btch: 1 usd: 4 2012-11-19 11:01:18 CPU 47: hi: 6, btch: 1 usd: 0 2012-11-19 11:01:18 active_anon:115999 inactive_anon:5012 isolated_anon:0 2012-11-19 11:01:18 active_file:65119 inactive_file:758589 isolated_file:0 2012-11-19 11:01:18 unevictable:0 dirty:223 writeback:64 unstable:0 2012-11-19 11:01:18 free:1079 slab_reclaimable:10755 slab_unreclaimable:16041 2012-11-19 11:01:18 mapped:1707 shmem:78 pagetables:615 bounce:0 2012-11-19 11:01:18 Node 0 DMA free:69056kB min:32128kB low:40128kB high:48192kB active_anon:7423936kB inactive_anon:320768kB active_file:4167616kB inactive_file:48549696kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:14272kB writeback:4096kB mapped:109248kB shmem:4992kB slab_reclaimable:688320kB slab_unreclaimable:1026624kB kernel_stack:26864kB pagetables:39360kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 11:01:18 lowmem_reserve[]: 0 0 0 2012-11-19 11:01:18 Node 0 DMA: 813*64kB 118*128kB 7*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 68928kB 2012-11-19 11:01:18 823179 total pagecache pages 2012-11-19 11:01:18 0 pages in swap cache 2012-11-19 11:01:18 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 11:01:18 Free swap = 4194176kB 2012-11-19 11:01:18 Total swap = 4194176kB 2012-11-19 11:01:18 1011712 pages RAM 2012-11-19 11:01:18 20742 pages reserved 2012-11-19 11:01:18 25922 pages shared 2012-11-19 11:01:18 973148 pages non-shared 2012-11-19 11:08:09 rpciod/8: page allocation failure. order:1, mode:0x20 2012-11-19 11:08:09 Call Trace: 2012-11-19 11:08:09 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 11:08:09 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 11:08:09 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 11:08:09 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 11:08:09 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 11:08:09 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 11:08:09 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 11:08:09 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 11:08:09 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 11:08:09 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 11:08:09 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 11:08:09 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 11:08:09 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 11:08:09 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 11:08:09 [c000000f498c6d00] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 11:08:09 [c000000f498c6da0] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 11:08:09 [c000000f498c6e20] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 11:08:09 [c000000f498c6ed0] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 11:08:09 --- Exception: 501 at .ip_queue_xmit+0x110/0x410 2012-11-19 11:08:09 LR = .ip_queue_xmit+0x78/0x410 2012-11-19 11:08:09 [c000000f498c72c0] [c000000000549fd4] .tcp_transmit_skb+0x474/0x9b0 2012-11-19 11:08:09 [c000000f498c73c0] [c00000000054d020] .tcp_write_xmit+0x230/0xbb0 2012-11-19 11:08:09 [c000000f498c74d0] [c00000000054da20] .__tcp_push_pending_frames+0x40/0x100 2012-11-19 11:08:09 [c000000f498c7550] [c00000000053d228] .tcp_sendmsg+0x2a8/0xdf0 2012-11-19 11:08:09 [c000000f498c76a0] [c0000000004ca3ec] .sock_sendmsg+0x14c/0x1a0 2012-11-19 11:08:09 [c000000f498c78c0] [c0000000004ca478] .kernel_sendmsg+0x38/0x60 2012-11-19 11:08:09 [c000000f498c7940] [d00000000bfe8ac4] .xs_send_kvec+0xb4/0xd0 [sunrpc] 2012-11-19 11:08:09 [c000000f498c7a00] [d00000000bfe8b9c] .xs_sendpages+0xbc/0x350 [sunrpc] 2012-11-19 11:08:09 [c000000f498c7ad0] [d00000000bfe8fdc] .xs_tcp_send_request+0x6c/0x1e0 [sunrpc] 2012-11-19 11:08:09 [c000000f498c7b80] [d00000000bfe59a4] .xprt_transmit+0xa4/0x340 [sunrpc] 2012-11-19 11:08:09 [c000000f498c7c40] [d00000000bfe1928] .call_transmit+0x1e8/0x2c0 [sunrpc] 2012-11-19 11:08:09 [c000000f498c7cd0] [d00000000bfef154] .__rpc_execute+0xc4/0x470 [sunrpc] 2012-11-19 11:08:09 [c000000f498c7db0] [c0000000000b6cac] .worker_thread+0x1dc/0x3c0 2012-11-19 11:08:09 [c000000f498c7ed0] [c0000000000bd02c] .kthread+0xbc/0xd0 2012-11-19 11:08:09 [c000000f498c7f90] [c000000000032054] .kernel_thread+0x54/0x70 2012-11-19 11:08:09 Mem-Info: 2012-11-19 11:08:09 Node 0 DMA per-cpu: 2012-11-19 11:08:09 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 10: hi: 6, btch: 1 usd: 1 2012-11-19 11:08:09 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 13: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 16: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 17: hi: 6, btch: 1 usd: 2 2012-11-19 11:08:09 CPU 18: hi: 6, btch: 1 usd: 3 2012-11-19 11:08:09 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 20: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 21: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 22: hi: 6, btch: 1 usd: 4 2012-11-19 11:08:09 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 24: hi: 6, btch: 1 usd: 3 2012-11-19 11:08:09 CPU 25: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 26: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 11:08:09 CPU 30: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 33: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 34: hi: 6, btch: 1 usd: 3 2012-11-19 11:08:09 CPU 35: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 36: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 37: hi: 6, btch: 1 usd: 1 2012-11-19 11:08:09 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 40: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 42: hi: 6, btch: 1 usd: 5 2012-11-19 11:08:09 CPU 43: hi: 6, btch: 1 usd: 3 2012-11-19 11:08:09 CPU 44: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 45: hi: 6, btch: 1 usd: 0 2012-11-19 11:08:09 CPU 46: hi: 6, btch: 1 usd: 1 2012-11-19 11:08:09 CPU 47: hi: 6, btch: 1 usd: 3 2012-11-19 11:08:09 active_anon:49418 inactive_anon:5012 isolated_anon:0 2012-11-19 11:08:09 active_file:59682 inactive_file:831904 isolated_file:32 2012-11-19 11:08:09 unevictable:0 dirty:237 writeback:0 unstable:0 2012-11-19 11:08:09 free:859 slab_reclaimable:9903 slab_unreclaimable:15915 2012-11-19 11:08:09 mapped:1650 shmem:78 pagetables:590 bounce:0 2012-11-19 11:08:09 Node 0 DMA free:55744kB min:32128kB low:40128kB high:48192kB active_anon:3162752kB inactive_anon:320768kB active_file:3819648kB inactive_file:53241856kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:15168kB writeback:0kB mapped:105600kB shmem:4992kB slab_reclaimable:633792kB slab_unreclaimable:1018560kB kernel_stack:26784kB pagetables:37760kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 11:08:09 lowmem_reserve[]: 0 0 0 2012-11-19 11:08:09 Node 0 DMA: 580*64kB 118*128kB 3*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 53504kB 2012-11-19 11:08:09 890702 total pagecache pages 2012-11-19 11:08:09 0 pages in swap cache 2012-11-19 11:08:09 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 11:08:09 Free swap = 4194176kB 2012-11-19 11:08:09 Total swap = 4194176kB 2012-11-19 11:08:09 1011712 pages RAM 2012-11-19 11:08:09 20742 pages reserved 2012-11-19 11:08:09 24341 pages shared 2012-11-19 11:08:09 975445 pages non-shared 2012-11-19 11:32:37 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 11:32:37 Call Trace: 2012-11-19 11:32:37 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 11:32:37 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 11:32:37 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 11:32:37 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 11:32:37 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 11:32:37 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 11:32:37 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 11:32:37 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 11:32:37 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 11:32:37 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 11:32:37 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 11:32:37 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 11:32:37 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 11:32:37 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 11:32:37 [c000000f59b7b970] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 11:32:37 [c000000f59b7ba10] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 11:32:37 [c000000f59b7ba90] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 11:32:37 --- Exception: 501 at 0x3b808cdca644 2012-11-19 11:32:37 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 11:32:37 [c000000f59b7bb40] [c000000000004898] hardware_interrupt_entry+0x18/0x80 (unreliable) 2012-11-19 11:32:37 --- Exception: 501 at .raw_local_irq_restore+0x34/0x60 2012-11-19 11:32:37 LR = .cpu_idle+0x14c/0x1d0 2012-11-19 11:32:37 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 (unreliable) 2012-11-19 11:32:37 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 11:32:37 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 11:32:37 Mem-Info: 2012-11-19 11:32:37 Node 0 DMA per-cpu: 2012-11-19 11:32:37 CPU 0: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 1: hi: 6, btch: 1 usd: 3 2012-11-19 11:32:37 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 4: hi: 6, btch: 1 usd: 2 2012-11-19 11:32:37 CPU 5: hi: 6, btch: 1 usd: 1 2012-11-19 11:32:37 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 12: hi: 6, btch: 1 usd: 2 2012-11-19 11:32:37 CPU 13: hi: 6, btch: 1 usd: 1 2012-11-19 11:32:37 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 15: hi: 6, btch: 1 usd: 4 2012-11-19 11:32:37 CPU 16: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 20: hi: 6, btch: 1 usd: 3 2012-11-19 11:32:37 CPU 21: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 24: hi: 6, btch: 1 usd: 3 2012-11-19 11:32:37 CPU 25: hi: 6, btch: 1 usd: 2 2012-11-19 11:32:37 CPU 26: hi: 6, btch: 1 usd: 1 2012-11-19 11:32:37 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 29: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 30: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 33: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 35: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 36: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 37: hi: 6, btch: 1 usd: 1 2012-11-19 11:32:37 CPU 38: hi: 6, btch: 1 usd: 1 2012-11-19 11:32:37 CPU 39: hi: 6, btch: 1 usd: 3 2012-11-19 11:32:37 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 42: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 44: hi: 6, btch: 1 usd: 5 2012-11-19 11:32:37 CPU 45: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 46: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 CPU 47: hi: 6, btch: 1 usd: 0 2012-11-19 11:32:37 active_anon:2998 inactive_anon:5044 isolated_anon:0 2012-11-19 11:32:37 active_file:39384 inactive_file:901726 isolated_file:32 2012-11-19 11:32:37 unevictable:0 dirty:469 writeback:12 unstable:0 2012-11-19 11:32:37 free:1036 slab_reclaimable:6228 slab_unreclaimable:16269 2012-11-19 11:32:37 mapped:1561 shmem:78 pagetables:565 bounce:0 2012-11-19 11:32:37 Node 0 DMA free:66304kB min:32128kB low:40128kB high:48192kB active_anon:191872kB inactive_anon:322816kB active_file:2520576kB inactive_file:57710464kB unevictable:0kB isolated(anon):0kB isolated(file):2048kB present:64694208kB mlocked:0kB dirty:30016kB writeback:768kB mapped:99904kB shmem:4992kB slab_reclaimable:398592kB slab_unreclaimable:1041216kB kernel_stack:26720kB pagetables:36160kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no 2012-11-19 11:32:37 lowmem_reserve[]: 0 0 0 2012-11-19 11:32:37 Node 0 DMA: 834*64kB 92*128kB 2*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 67712kB 2012-11-19 11:32:37 941103 total pagecache pages 2012-11-19 11:32:37 0 pages in swap cache 2012-11-19 11:32:37 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 11:32:37 Free swap = 4194176kB 2012-11-19 11:32:37 Total swap = 4194176kB 2012-11-19 11:32:37 1011712 pages RAM 2012-11-19 11:32:37 20742 pages reserved 2012-11-19 11:32:37 84919 pages shared 2012-11-19 11:32:37 921997 pages non-shared 2012-11-19 11:36:16 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 11:36:16 Call Trace: 2012-11-19 11:36:16 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 11:36:16 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 11:36:16 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 11:36:16 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 11:36:16 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 11:36:16 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 11:36:16 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 11:36:16 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 11:36:16 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 11:36:16 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 11:36:16 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 11:36:16 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 11:36:16 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 11:36:16 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 11:36:16 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 11:36:16 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 11:36:16 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 11:36:16 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 11:36:16 --- Exception: 501 at .pseries_dedicated_idle_sleep+0xf0/0x1f0 2012-11-19 11:36:16 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 11:36:16 [c000000f59b7bd80] [0001db8d5608164b] 0x1db8d5608164b (unreliable) 2012-11-19 11:36:16 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 11:36:16 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 11:36:16 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 11:36:16 Mem-Info: 2012-11-19 11:36:16 Node 0 DMA per-cpu: 2012-11-19 11:36:16 CPU 0: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:16 CPU 1: hi: 6, btch: 1 usd: 3 2012-11-19 11:36:16 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 5: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:16 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 7: hi: 6, btch: 1 usd: 1 2012-11-19 11:36:16 CPU 8: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:16 CPU 9: hi: 6, btch: 1 usd: 2 2012-11-19 11:36:16 CPU 10: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 16: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 17: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:16 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 24: hi: 6, btch: 1 usd: 2 2012-11-19 11:36:16 CPU 25: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:16 CPU 26: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 28: hi: 6, btch: 1 usd: 2 2012-11-19 11:36:16 CPU 29: hi: 6, btch: 1 usd: 2 2012-11-19 11:36:16 CPU 30: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 32: hi: 6, btch: 1 usd: 1 2012-11-19 11:36:16 CPU 33: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:16 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 35: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 36: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:16 CPU 37: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 41: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 42: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 44: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 45: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 46: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 47: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 active_anon:4028 inactive_anon:5045 isolated_anon:0 2012-11-19 11:36:16 active_file:38899 inactive_file:902057 isolated_file:0 2012-11-19 11:36:16 unevictable:0 dirty:887 writeback:0 unstable:0 2012-11-19 11:36:16 free:652 slab_reclaimable:6110 slab_unreclaimable:15876 2012-11-19 11:36:16 mapped:1586 shmem:79 pagetables:581 bounce:0 2012-11-19 11:36:16 Node 0 DMA free:41728kB min:32128kB low:40128kB high:48192kB active_anon:257792kB inactive_anon:322880kB active_file:2489536kB inactive_file:57731648kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:56768kB writeback:0kB mapped:101504kB shmem:5056kB slab_reclaimable:391040kB slab_unreclaimable:1016064kB kernel_stack:26864kB pagetables:37184kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 11:36:16 lowmem_reserve[]: 0 0 0 2012-11-19 11:36:16 Node 0 DMA: 561*64kB 16*128kB 6*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 41536kB 2012-11-19 11:36:16 941153 total pagecache pages 2012-11-19 11:36:16 0 pages in swap cache 2012-11-19 11:36:16 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 11:36:16 Free swap = 4194176kB 2012-11-19 11:36:16 Total swap = 4194176kB 2012-11-19 11:36:16 1011712 pages RAM 2012-11-19 11:36:16 20742 pages reserved 2012-11-19 11:36:16 50747 pages shared 2012-11-19 11:36:16 948785 pages non-shared 2012-11-19 11:36:16 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 11:36:16 Call Trace: 2012-11-19 11:36:16 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 11:36:16 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 11:36:16 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 11:36:16 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 11:36:16 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 11:36:16 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 11:36:16 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 11:36:16 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 11:36:16 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 11:36:16 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 11:36:16 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 11:36:16 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 11:36:16 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 11:36:16 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 11:36:16 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 11:36:16 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 11:36:16 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 11:36:16 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 11:36:16 --- Exception: 501 at .pseries_dedicated_idle_sleep+0xf0/0x1f0 2012-11-19 11:36:16 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 11:36:16 [c000000f59b7bd80] [0001db8d5608164b] 0x1db8d5608164b (unreliable) 2012-11-19 11:36:16 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 11:36:16 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 11:36:16 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 11:36:16 Mem-Info: 2012-11-19 11:36:16 Node 0 DMA per-cpu: 2012-11-19 11:36:16 CPU 0: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:16 CPU 1: hi: 6, btch: 1 usd: 3 2012-11-19 11:36:16 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 5: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:16 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 7: hi: 6, btch: 1 usd: 1 2012-11-19 11:36:16 CPU 8: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:16 CPU 9: hi: 6, btch: 1 usd: 2 2012-11-19 11:36:16 CPU 10: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 16: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 17: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:16 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 24: hi: 6, btch: 1 usd: 2 2012-11-19 11:36:16 CPU 25: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:16 CPU 26: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 28: hi: 6, btch: 1 usd: 2 2012-11-19 11:36:16 CPU 29: hi: 6, btch: 1 usd: 2 2012-11-19 11:36:16 CPU 30: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:16 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 32: hi: 6, btch: 1 usd: 1 2012-11-19 11:36:16 CPU 33: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:16 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 35: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 36: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:16 CPU 37: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 40: hi: 6, btch: 1 usd: 1 2012-11-19 11:36:16 CPU 41: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 42: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 44: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 45: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 46: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 CPU 47: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:16 active_anon:4184 inactive_anon:5045 isolated_anon:0 2012-11-19 11:36:16 active_file:38899 inactive_file:901941 isolated_file:0 2012-11-19 11:36:16 unevictable:0 dirty:891 writeback:0 unstable:0 2012-11-19 11:36:16 free:634 slab_reclaimable:6110 slab_unreclaimable:15876 2012-11-19 11:36:16 mapped:1586 shmem:79 pagetables:581 bounce:0 2012-11-19 11:36:16 Node 0 DMA free:40576kB min:32128kB low:40128kB high:48192kB active_anon:267776kB inactive_anon:322880kB active_file:2489536kB inactive_file:57724224kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:57024kB writeback:0kB mapped:101504kB shmem:5056kB slab_reclaimable:391040kB slab_unreclaimable:1016064kB kernel_stack:26864kB pagetables:37184kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 11:36:16 lowmem_reserve[]: 0 0 0 2012-11-19 11:36:16 Node 0 DMA: 542*64kB 16*128kB 6*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 40320kB 2012-11-19 11:36:16 941037 total pagecache pages 2012-11-19 11:36:16 0 pages in swap cache 2012-11-19 11:36:16 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 11:36:16 Free swap = 4194176kB 2012-11-19 11:36:16 Total swap = 4194176kB 2012-11-19 11:36:16 1011712 pages RAM 2012-11-19 11:36:16 20742 pages reserved 2012-11-19 11:36:16 50258 pages shared 2012-11-19 11:36:16 948684 pages non-shared 2012-11-19 11:36:18 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 11:36:18 Call Trace: 2012-11-19 11:36:18 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 11:36:18 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 11:36:18 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 11:36:18 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 11:36:18 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 11:36:18 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 11:36:18 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 11:36:18 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 11:36:18 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 11:36:18 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 11:36:18 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 11:36:18 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 11:36:18 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 11:36:18 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 11:36:18 [c000000f59b7b970] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 11:36:18 [c000000f59b7ba10] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 11:36:18 [c000000f59b7ba90] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 11:36:18 --- Exception: 501 at 0x3b867925d7bc 2012-11-19 11:36:18 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 11:36:18 [c000000f59b7bb40] [c000000000004898] hardware_interrupt_entry+0x18/0x80 (unreliable) 2012-11-19 11:36:18 --- Exception: 501 at .raw_local_irq_restore+0x34/0x60 2012-11-19 11:36:18 LR = .cpu_idle+0x14c/0x1d0 2012-11-19 11:36:18 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 (unreliable) 2012-11-19 11:36:18 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 11:36:18 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 11:36:18 Mem-Info: 2012-11-19 11:36:18 Node 0 DMA per-cpu: 2012-11-19 11:36:18 CPU 0: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 5: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:18 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 7: hi: 6, btch: 1 usd: 1 2012-11-19 11:36:18 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 9: hi: 6, btch: 1 usd: 2 2012-11-19 11:36:18 CPU 10: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 16: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 17: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:18 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 24: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 25: hi: 6, btch: 1 usd: 3 2012-11-19 11:36:18 CPU 26: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 29: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 30: hi: 6, btch: 1 usd: 5 2012-11-19 11:36:18 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 32: hi: 6, btch: 1 usd: 1 2012-11-19 11:36:18 CPU 33: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:18 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 35: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 36: hi: 6, btch: 1 usd: 4 2012-11-19 11:36:18 CPU 37: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 41: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 42: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 44: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 45: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 46: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 CPU 47: hi: 6, btch: 1 usd: 0 2012-11-19 11:36:18 active_anon:3088 inactive_anon:5045 isolated_anon:0 2012-11-19 11:36:18 active_file:38881 inactive_file:902684 isolated_file:32 2012-11-19 11:36:18 unevictable:0 dirty:825 writeback:0 unstable:0 2012-11-19 11:36:18 free:973 slab_reclaimable:6106 slab_unreclaimable:15897 2012-11-19 11:36:18 mapped:1563 shmem:79 pagetables:573 bounce:0 2012-11-19 11:36:18 Node 0 DMA free:63424kB min:32128kB low:40128kB high:48192kB active_anon:197632kB inactive_anon:322880kB active_file:2488384kB inactive_file:57771776kB unevictable:0kB isolated(anon):0kB isolated(file):2048kB present:64694208kB mlocked:0kB dirty:52800kB writeback:0kB mapped:100032kB shmem:5056kB slab_reclaimable:390784kB slab_unreclaimable:1017408kB kernel_stack:26800kB pagetables:36672kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 11:36:18 lowmem_reserve[]: 0 0 0 2012-11-19 11:36:18 Node 0 DMA: 725*64kB 117*128kB 6*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 64960kB 2012-11-19 11:36:18 941752 total pagecache pages 2012-11-19 11:36:18 0 pages in swap cache 2012-11-19 11:36:18 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 11:36:18 Free swap = 4194176kB 2012-11-19 11:36:18 Total swap = 4194176kB 2012-11-19 11:36:18 1011712 pages RAM 2012-11-19 11:36:18 20742 pages reserved 2012-11-19 11:36:18 52541 pages shared 2012-11-19 11:36:18 946426 pages non-shared 2012-11-19 11:37:55 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 11:37:55 Call Trace: 2012-11-19 11:37:55 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 11:37:55 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 11:37:55 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 11:37:55 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 11:37:55 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 11:37:55 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 11:37:55 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 11:37:55 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 11:37:55 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 11:37:55 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 11:37:55 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 11:37:55 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 11:37:55 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 11:37:55 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 11:37:55 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 11:37:55 --- Exception: 501 at .ppc64_runlatch_off+0x24/0x60 2012-11-19 11:37:55 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 11:37:55 [c000000f59b7bd80] [0001dba45a47e3aa] 0x1dba45a47e3aa (unreliable) 2012-11-19 11:37:55 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 11:37:55 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 11:37:55 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 11:37:55 Mem-Info: 2012-11-19 11:37:55 Node 0 DMA per-cpu: 2012-11-19 11:37:55 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 1: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 11:37:55 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 11:37:55 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 24: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 25: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 26: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 27: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 28: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 30: hi: 6, btch: 1 usd: 1 2012-11-19 11:37:55 CPU 31: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 34: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 35: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 36: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 37: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 38: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 39: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 40: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 41: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 42: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 43: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 44: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 active_anon:3333 inactive_anon:5045 isolated_anon:0 2012-11-19 11:37:55 active_file:31542 inactive_file:910055 isolated_file:0 2012-11-19 11:37:55 unevictable:0 dirty:240 writeback:0 unstable:0 2012-11-19 11:37:55 free:719 slab_reclaimable:5992 slab_unreclaimable:15944 2012-11-19 11:37:55 mapped:1565 shmem:79 pagetables:576 bounce:0 2012-11-19 11:37:55 Node 0 DMA free:46016kB min:32128kB low:40128kB high:48192kB active_anon:213312kB inactive_anon:322880kB active_file:2018688kB inactive_file:58243520kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:15360kB writeback:0kB mapped:100160kB shmem:5056kB slab_reclaimable:383488kB slab_unreclaimable:1020416kB kernel_stack:26864kB pagetables:36864kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 11:37:55 lowmem_reserve[]: 0 0 0 2012-11-19 11:37:55 Node 0 DMA: 625*64kB 17*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 46016kB 2012-11-19 11:37:55 941811 total pagecache pages 2012-11-19 11:37:55 0 pages in swap cache 2012-11-19 11:37:55 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 11:37:55 Free swap = 4194176kB 2012-11-19 11:37:55 Total swap = 4194176kB 2012-11-19 11:37:55 1011712 pages RAM 2012-11-19 11:37:55 20742 pages reserved 2012-11-19 11:37:55 59326 pages shared 2012-11-19 11:37:55 943424 pages non-shared 2012-11-19 11:37:55 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 11:37:55 Call Trace: 2012-11-19 11:37:55 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 11:37:55 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 11:37:55 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 11:37:55 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 11:37:55 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 11:37:55 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 11:37:55 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 11:37:55 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 11:37:55 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 11:37:55 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 11:37:55 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 11:37:55 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 11:37:55 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 11:37:55 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 11:37:55 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 11:37:55 --- Exception: 501 at .ppc64_runlatch_off+0x24/0x60 2012-11-19 11:37:55 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 11:37:55 [c000000f59b7bd80] [0001dba45a47e3aa] 0x1dba45a47e3aa (unreliable) 2012-11-19 11:37:55 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 11:37:55 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 11:37:55 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 11:37:55 Mem-Info: 2012-11-19 11:37:55 Node 0 DMA per-cpu: 2012-11-19 11:37:55 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 1: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 11:37:55 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 11:37:55 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 24: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 25: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 26: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 27: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 28: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 30: hi: 6, btch: 1 usd: 1 2012-11-19 11:37:55 CPU 31: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 34: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 35: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 36: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 37: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 38: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 39: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 40: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 41: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 42: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 43: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 44: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 active_anon:3333 inactive_anon:5045 isolated_anon:0 2012-11-19 11:37:55 active_file:31542 inactive_file:910055 isolated_file:0 2012-11-19 11:37:55 unevictable:0 dirty:248 writeback:0 unstable:0 2012-11-19 11:37:55 free:713 slab_reclaimable:5992 slab_unreclaimable:15944 2012-11-19 11:37:55 mapped:1565 shmem:79 pagetables:576 bounce:0 2012-11-19 11:37:55 Node 0 DMA free:45632kB min:32128kB low:40128kB high:48192kB active_anon:213312kB inactive_anon:322880kB active_file:2018688kB inactive_file:58243520kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:15872kB writeback:0kB mapped:100160kB shmem:5056kB slab_reclaimable:383488kB slab_unreclaimable:1020416kB kernel_stack:26864kB pagetables:36864kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 11:37:55 lowmem_reserve[]: 0 0 0 2012-11-19 11:37:55 Node 0 DMA: 617*64kB 17*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 45504kB 2012-11-19 11:37:55 941819 total pagecache pages 2012-11-19 11:37:55 0 pages in swap cache 2012-11-19 11:37:55 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 11:37:55 Free swap = 4194176kB 2012-11-19 11:37:55 Total swap = 4194176kB 2012-11-19 11:37:55 1011712 pages RAM 2012-11-19 11:37:55 20742 pages reserved 2012-11-19 11:37:55 59326 pages shared 2012-11-19 11:37:55 943424 pages non-shared 2012-11-19 11:37:55 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 11:37:55 Call Trace: 2012-11-19 11:37:55 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 11:37:55 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 11:37:55 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 11:37:55 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 11:37:55 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 11:37:55 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 11:37:55 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 11:37:55 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 11:37:55 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 11:37:55 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 11:37:55 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 11:37:55 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 11:37:55 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 11:37:55 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 11:37:55 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 11:37:55 --- Exception: 501 at .ppc64_runlatch_off+0x24/0x60 2012-11-19 11:37:55 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 11:37:55 [c000000f59b7bd80] [0001dba45a47e3aa] 0x1dba45a47e3aa (unreliable) 2012-11-19 11:37:55 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 11:37:55 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 11:37:55 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 11:37:55 Mem-Info: 2012-11-19 11:37:55 Node 0 DMA per-cpu: 2012-11-19 11:37:55 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 1: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 11:37:55 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 11:37:55 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 24: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 25: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 26: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 27: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 28: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 30: hi: 6, btch: 1 usd: 1 2012-11-19 11:37:55 CPU 31: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 34: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 35: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 36: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 37: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 38: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 39: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 40: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 41: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 42: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 43: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 44: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 active_anon:3333 inactive_anon:5045 isolated_anon:0 2012-11-19 11:37:55 active_file:31542 inactive_file:910055 isolated_file:0 2012-11-19 11:37:55 unevictable:0 dirty:246 writeback:0 unstable:0 2012-11-19 11:37:55 free:713 slab_reclaimable:5992 slab_unreclaimable:15944 2012-11-19 11:37:55 mapped:1565 shmem:79 pagetables:576 bounce:0 2012-11-19 11:37:55 Node 0 DMA free:45632kB min:32128kB low:40128kB high:48192kB active_anon:213312kB inactive_anon:322880kB active_file:2018688kB inactive_file:58243520kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:15744kB writeback:0kB mapped:100160kB shmem:5056kB slab_reclaimable:383488kB slab_unreclaimable:1020416kB kernel_stack:26864kB pagetables:36864kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 11:37:55 lowmem_reserve[]: 0 0 0 2012-11-19 11:37:55 Node 0 DMA: 617*64kB 17*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 45504kB 2012-11-19 11:37:55 941819 total pagecache pages 2012-11-19 11:37:55 0 pages in swap cache 2012-11-19 11:37:55 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 11:37:55 Free swap = 4194176kB 2012-11-19 11:37:55 Total swap = 4194176kB 2012-11-19 11:37:55 1011712 pages RAM 2012-11-19 11:37:55 20742 pages reserved 2012-11-19 11:37:55 59326 pages shared 2012-11-19 11:37:55 943424 pages non-shared 2012-11-19 11:37:55 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 11:37:55 Call Trace: 2012-11-19 11:37:55 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 11:37:55 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 11:37:55 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 11:37:55 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 11:37:55 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 11:37:55 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 11:37:55 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 11:37:55 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 11:37:55 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 11:37:55 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 11:37:55 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 11:37:55 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 11:37:55 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 11:37:55 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 11:37:55 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 11:37:55 --- Exception: 501 at .ppc64_runlatch_off+0x24/0x60 2012-11-19 11:37:55 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 11:37:55 [c000000f59b7bd80] [0001dba45a47e3aa] 0x1dba45a47e3aa (unreliable) 2012-11-19 11:37:55 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 11:37:55 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 11:37:55 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 11:37:55 Mem-Info: 2012-11-19 11:37:55 Node 0 DMA per-cpu: 2012-11-19 11:37:55 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 1: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 11:37:55 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 11:37:55 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 24: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 25: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 26: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 27: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 28: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 30: hi: 6, btch: 1 usd: 1 2012-11-19 11:37:55 CPU 31: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 34: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 35: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 36: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 37: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 38: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 39: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 40: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 41: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 42: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 43: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 44: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 active_anon:3333 inactive_anon:5045 isolated_anon:0 2012-11-19 11:37:55 active_file:31542 inactive_file:910055 isolated_file:0 2012-11-19 11:37:55 unevictable:0 dirty:246 writeback:0 unstable:0 2012-11-19 11:37:55 free:713 slab_reclaimable:5992 slab_unreclaimable:15944 2012-11-19 11:37:55 mapped:1565 shmem:79 pagetables:576 bounce:0 2012-11-19 11:37:55 Node 0 DMA free:45632kB min:32128kB low:40128kB high:48192kB active_anon:213312kB inactive_anon:322880kB active_file:2018688kB inactive_file:58243520kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:15744kB writeback:0kB mapped:100160kB shmem:5056kB slab_reclaimable:383488kB slab_unreclaimable:1020416kB kernel_stack:26864kB pagetables:36864kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 11:37:55 lowmem_reserve[]: 0 0 0 2012-11-19 11:37:55 Node 0 DMA: 617*64kB 17*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 45504kB 2012-11-19 11:37:55 941819 total pagecache pages 2012-11-19 11:37:55 0 pages in swap cache 2012-11-19 11:37:55 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 11:37:55 Free swap = 4194176kB 2012-11-19 11:37:55 Total swap = 4194176kB 2012-11-19 11:37:55 1011712 pages RAM 2012-11-19 11:37:55 20742 pages reserved 2012-11-19 11:37:55 59326 pages shared 2012-11-19 11:37:55 943424 pages non-shared 2012-11-19 11:37:55 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 11:37:55 Call Trace: 2012-11-19 11:37:55 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 11:37:55 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 11:37:55 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 11:37:55 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 11:37:55 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 11:37:55 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 11:37:55 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 11:37:55 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 11:37:55 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 11:37:55 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 11:37:55 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 11:37:55 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 11:37:55 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 11:37:55 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 11:37:55 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 11:37:55 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 11:37:55 --- Exception: 501 at .ppc64_runlatch_off+0x24/0x60 2012-11-19 11:37:55 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 11:37:55 [c000000f59b7bd80] [0001dba45a47e3aa] 0x1dba45a47e3aa (unreliable) 2012-11-19 11:37:55 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 11:37:55 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 11:37:55 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 11:37:55 Mem-Info: 2012-11-19 11:37:55 Node 0 DMA per-cpu: 2012-11-19 11:37:55 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 11:37:55 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 11:37:55 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 24: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 25: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 26: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 27: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 28: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 29: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 30: hi: 6, btch: 1 usd: 1 2012-11-19 11:37:55 CPU 31: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 34: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 35: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 36: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 37: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 38: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 39: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 11:37:55 CPU 41: hi: 6, btch: 1 usd: 2 2012-11-19 11:37:55 CPU 42: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 43: hi: 6, btch: 1 usd: 3 2012-11-19 11:37:55 CPU 44: hi: 6, btch: 1 usd: 4 2012-11-19 11:37:55 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 11:37:55 active_anon:3331 inactive_anon:5045 isolated_anon:0 2012-11-19 11:37:55 active_file:31542 inactive_file:910055 isolated_file:0 2012-11-19 11:37:55 unevictable:0 dirty:230 writeback:0 unstable:0 2012-11-19 11:37:55 free:712 slab_reclaimable:5991 slab_unreclaimable:15944 2012-11-19 11:37:55 mapped:1565 shmem:79 pagetables:576 bounce:0 2012-11-19 11:37:55 Node 0 DMA free:45568kB min:32128kB low:40128kB high:48192kB active_anon:213184kB inactive_anon:322880kB active_file:2018688kB inactive_file:58243520kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:14720kB writeback:0kB mapped:100160kB shmem:5056kB slab_reclaimable:383424kB slab_unreclaimable:1020416kB kernel_stack:26864kB pagetables:36864kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 11:37:55 lowmem_reserve[]: 0 0 0 2012-11-19 11:37:55 Node 0 DMA: 616*64kB 17*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 45440kB 2012-11-19 11:37:55 941819 total pagecache pages 2012-11-19 11:37:55 0 pages in swap cache 2012-11-19 11:37:55 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 11:37:55 Free swap = 4194176kB 2012-11-19 11:37:55 Total swap = 4194176kB 2012-11-19 11:37:55 1011712 pages RAM 2012-11-19 11:37:55 20742 pages reserved 2012-11-19 11:37:55 59330 pages shared 2012-11-19 11:37:55 943425 pages non-shared 2012-11-19 11:38:25 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 11:38:25 Call Trace: 2012-11-19 11:38:25 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 11:38:25 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 11:38:25 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 11:38:25 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 11:38:25 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 11:38:25 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 11:38:25 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 11:38:25 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 11:38:25 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 11:38:25 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 11:38:25 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 11:38:25 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 11:38:25 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 11:38:25 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 11:38:25 [c000000f59b7b970] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 11:38:25 [c000000f59b7ba10] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 11:38:25 [c000000f59b7ba90] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 11:38:25 --- Exception: 501 at 0x3b89db310aec 2012-11-19 11:38:25 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 11:38:25 [c000000f59b7bb40] [c000000000004898] hardware_interrupt_entry+0x18/0x80 (unreliable) 2012-11-19 11:38:25 --- Exception: 501 at .raw_local_irq_restore+0x34/0x60 2012-11-19 11:38:25 LR = .cpu_idle+0x14c/0x1d0 2012-11-19 11:38:25 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 (unreliable) 2012-11-19 11:38:25 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 11:38:25 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 11:38:25 Mem-Info: 2012-11-19 11:38:25 Node 0 DMA per-cpu: 2012-11-19 11:38:25 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 11:38:25 CPU 10: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 16: hi: 6, btch: 1 usd: 0 2012-11-19 11:38:25 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 24: hi: 6, btch: 1 usd: 4 2012-11-19 11:38:25 CPU 25: hi: 6, btch: 1 usd: 4 2012-11-19 11:38:25 CPU 26: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 27: hi: 6, btch: 1 usd: 3 2012-11-19 11:38:25 CPU 28: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 29: hi: 6, btch: 1 usd: 1 2012-11-19 11:38:25 CPU 30: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 31: hi: 6, btch: 1 usd: 3 2012-11-19 11:38:25 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 33: hi: 6, btch: 1 usd: 0 2012-11-19 11:38:25 CPU 34: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 35: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 36: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 37: hi: 6, btch: 1 usd: 0 2012-11-19 11:38:25 CPU 38: hi: 6, btch: 1 usd: 3 2012-11-19 11:38:25 CPU 39: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 11:38:25 CPU 41: hi: 6, btch: 1 usd: 4 2012-11-19 11:38:25 CPU 42: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 44: hi: 6, btch: 1 usd: 3 2012-11-19 11:38:25 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 11:38:25 active_anon:3332 inactive_anon:5045 isolated_anon:0 2012-11-19 11:38:25 active_file:30371 inactive_file:910728 isolated_file:33 2012-11-19 11:38:25 unevictable:0 dirty:38 writeback:0 unstable:0 2012-11-19 11:38:25 free:1273 slab_reclaimable:5895 slab_unreclaimable:15952 2012-11-19 11:38:25 mapped:1566 shmem:79 pagetables:577 bounce:0 2012-11-19 11:38:25 Node 0 DMA free:82624kB min:32128kB low:40128kB high:48192kB active_anon:213248kB inactive_anon:322880kB active_file:1943744kB inactive_file:58286592kB unevictable:0kB isolated(anon):0kB isolated(file):2112kB present:64694208kB mlocked:0kB dirty:2432kB writeback:0kB mapped:100224kB shmem:5056kB slab_reclaimable:377280kB slab_unreclaimable:1020928kB kernel_stack:26848kB pagetables:36928kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 11:38:25 lowmem_reserve[]: 0 0 0 2012-11-19 11:38:25 Node 0 DMA: 1010*64kB 121*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 83968kB 2012-11-19 11:38:25 940988 total pagecache pages 2012-11-19 11:38:25 0 pages in swap cache 2012-11-19 11:38:25 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 11:38:25 Free swap = 4194176kB 2012-11-19 11:38:25 Total swap = 4194176kB 2012-11-19 11:38:25 1011712 pages RAM 2012-11-19 11:38:25 20742 pages reserved 2012-11-19 11:38:25 54689 pages shared 2012-11-19 11:38:25 939187 pages non-shared Console [seqlac2] log at 2012-11-19 12:00:00 PST. Console [seqlac2] log at 2012-11-19 13:00:00 PST. Console [seqlac2] log at 2012-11-19 14:00:00 PST. 2012-11-19 14:42:31 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 14:42:31 Call Trace: 2012-11-19 14:42:31 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 14:42:31 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 14:42:31 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 14:42:31 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 14:42:31 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 14:42:31 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 14:42:31 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 14:42:31 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 14:42:31 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 14:42:31 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 14:42:31 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 14:42:31 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 14:42:31 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 14:42:31 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 14:42:31 [c000000f59b7b970] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 14:42:31 [c000000f59b7ba10] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 14:42:31 [c000000f59b7ba90] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 14:42:31 [c000000f59b7bb40] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 14:42:31 --- Exception: 501 at .raw_local_irq_restore+0x34/0x60 2012-11-19 14:42:31 LR = .cpu_idle+0x14c/0x1d0 2012-11-19 14:42:31 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 (unreliable) 2012-11-19 14:42:31 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 14:42:31 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 14:42:31 Mem-Info: 2012-11-19 14:42:31 Node 0 DMA per-cpu: 2012-11-19 14:42:31 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 1: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 2: hi: 6, btch: 1 usd: 4 2012-11-19 14:42:31 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 4: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 5: hi: 6, btch: 1 usd: 1 2012-11-19 14:42:31 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 8: hi: 6, btch: 1 usd: 3 2012-11-19 14:42:31 CPU 9: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 10: hi: 6, btch: 1 usd: 1 2012-11-19 14:42:31 CPU 11: hi: 6, btch: 1 usd: 3 2012-11-19 14:42:31 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 13: hi: 6, btch: 1 usd: 2 2012-11-19 14:42:31 CPU 14: hi: 6, btch: 1 usd: 2 2012-11-19 14:42:31 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 17: hi: 6, btch: 1 usd: 2 2012-11-19 14:42:31 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 21: hi: 6, btch: 1 usd: 3 2012-11-19 14:42:31 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 24: hi: 6, btch: 1 usd: 2 2012-11-19 14:42:31 CPU 25: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 26: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 29: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 30: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 33: hi: 6, btch: 1 usd: 1 2012-11-19 14:42:31 CPU 34: hi: 6, btch: 1 usd: 2 2012-11-19 14:42:31 CPU 35: hi: 6, btch: 1 usd: 2 2012-11-19 14:42:31 CPU 36: hi: 6, btch: 1 usd: 4 2012-11-19 14:42:31 CPU 37: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 41: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 42: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 44: hi: 6, btch: 1 usd: 4 2012-11-19 14:42:31 CPU 45: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 46: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 47: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 active_anon:4695 inactive_anon:4954 isolated_anon:0 2012-11-19 14:42:31 active_file:13212 inactive_file:932842 isolated_file:0 2012-11-19 14:42:31 unevictable:0 dirty:170 writeback:0 unstable:0 2012-11-19 14:42:31 free:534 slab_reclaimable:2153 slab_unreclaimable:14004 2012-11-19 14:42:31 mapped:1690 shmem:80 pagetables:729 bounce:0 2012-11-19 14:42:31 Node 0 DMA free:34176kB min:32128kB low:40128kB high:48192kB active_anon:300480kB inactive_anon:317056kB active_file:845568kB inactive_file:59701888kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:10880kB writeback:0kB mapped:108160kB shmem:5120kB slab_reclaimable:137792kB slab_unreclaimable:896256kB kernel_stack:26960kB pagetables:46656kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 14:42:31 lowmem_reserve[]: 0 0 0 2012-11-19 14:42:31 Node 0 DMA: 533*64kB 17*128kB 5*256kB 4*512kB 1*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 42688kB 2012-11-19 14:42:31 946289 total pagecache pages 2012-11-19 14:42:31 0 pages in swap cache 2012-11-19 14:42:31 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 14:42:31 Free swap = 4194176kB 2012-11-19 14:42:31 Total swap = 4194176kB 2012-11-19 14:42:31 1011712 pages RAM 2012-11-19 14:42:31 20742 pages reserved 2012-11-19 14:42:31 26464 pages shared 2012-11-19 14:42:31 974327 pages non-shared 2012-11-19 14:42:31 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 14:42:31 Call Trace: 2012-11-19 14:42:31 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 14:42:31 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 14:42:31 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 14:42:31 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 14:42:31 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 14:42:31 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 14:42:31 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 14:42:31 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 14:42:31 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 14:42:31 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 14:42:31 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 14:42:31 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 14:42:31 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 14:42:31 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 14:42:31 [c000000f59b7b970] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 14:42:31 [c000000f59b7ba10] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 14:42:31 [c000000f59b7ba90] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 14:42:31 [c000000f59b7bb40] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 14:42:31 --- Exception: 501 at .raw_local_irq_restore+0x34/0x60 2012-11-19 14:42:31 LR = .cpu_idle+0x14c/0x1d0 2012-11-19 14:42:31 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 (unreliable) 2012-11-19 14:42:31 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 14:42:31 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 14:42:31 Mem-Info: 2012-11-19 14:42:31 Node 0 DMA per-cpu: 2012-11-19 14:42:31 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 1: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 4: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 5: hi: 6, btch: 1 usd: 1 2012-11-19 14:42:31 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 8: hi: 6, btch: 1 usd: 3 2012-11-19 14:42:31 CPU 9: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 10: hi: 6, btch: 1 usd: 1 2012-11-19 14:42:31 CPU 11: hi: 6, btch: 1 usd: 3 2012-11-19 14:42:31 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 13: hi: 6, btch: 1 usd: 2 2012-11-19 14:42:31 CPU 14: hi: 6, btch: 1 usd: 2 2012-11-19 14:42:31 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 17: hi: 6, btch: 1 usd: 2 2012-11-19 14:42:31 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 21: hi: 6, btch: 1 usd: 3 2012-11-19 14:42:31 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 24: hi: 6, btch: 1 usd: 2 2012-11-19 14:42:31 CPU 25: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 26: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 28: hi: 6, btch: 1 usd: 1 2012-11-19 14:42:31 CPU 29: hi: 6, btch: 1 usd: 5 2012-11-19 14:42:31 CPU 30: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 33: hi: 6, btch: 1 usd: 1 2012-11-19 14:42:31 CPU 34: hi: 6, btch: 1 usd: 2 2012-11-19 14:42:31 CPU 35: hi: 6, btch: 1 usd: 2 2012-11-19 14:42:31 CPU 36: hi: 6, btch: 1 usd: 4 2012-11-19 14:42:31 CPU 37: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 40: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 41: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 42: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 44: hi: 6, btch: 1 usd: 4 2012-11-19 14:42:31 CPU 45: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 46: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 CPU 47: hi: 6, btch: 1 usd: 0 2012-11-19 14:42:31 active_anon:5520 inactive_anon:4954 isolated_anon:0 2012-11-19 14:42:31 active_file:13217 inactive_file:931879 isolated_file:0 2012-11-19 14:42:31 unevictable:0 dirty:170 writeback:0 unstable:0 2012-11-19 14:42:31 free:675 slab_reclaimable:2153 slab_unreclaimable:14004 2012-11-19 14:42:31 mapped:1690 shmem:80 pagetables:729 bounce:0 2012-11-19 14:42:31 Node 0 DMA free:43200kB min:32128kB low:40128kB high:48192kB active_anon:353280kB inactive_anon:317056kB active_file:845888kB inactive_file:59640256kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:10880kB writeback:0kB mapped:108160kB shmem:5120kB slab_reclaimable:137792kB slab_unreclaimable:896256kB kernel_stack:26960kB pagetables:46656kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 14:42:31 lowmem_reserve[]: 0 0 0 2012-11-19 14:42:31 Node 0 DMA: 678*64kB 17*128kB 5*256kB 4*512kB 1*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 51968kB 2012-11-19 14:42:31 945329 total pagecache pages 2012-11-19 14:42:31 0 pages in swap cache 2012-11-19 14:42:31 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 14:42:31 Free swap = 4194176kB 2012-11-19 14:42:31 Total swap = 4194176kB 2012-11-19 14:42:31 1011712 pages RAM 2012-11-19 14:42:31 20742 pages reserved 2012-11-19 14:42:31 25954 pages shared 2012-11-19 14:42:31 973732 pages non-shared Console [seqlac2] log at 2012-11-19 15:00:00 PST. 2012-11-19 15:15:50 rpciod/8: page allocation failure. order:1, mode:0x20 2012-11-19 15:15:50 Call Trace: 2012-11-19 15:15:50 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 15:15:50 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 15:15:50 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 15:15:50 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 15:15:50 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 15:15:50 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 15:15:50 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 15:15:50 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 15:15:50 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 15:15:50 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 15:15:50 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 15:15:50 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 15:15:50 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 15:15:50 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 15:15:50 [c000000f498c7770] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 15:15:50 [c000000f498c7810] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 15:15:50 [c000000f498c7890] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 15:15:50 [c000000f498c7940] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 15:15:50 --- Exception: 501 at .call_status+0x40/0x2a0 [sunrpc] 2012-11-19 15:15:50 LR = .__rpc_execute+0xc4/0x470 [sunrpc] 2012-11-19 15:15:50 [c000000f498c7c30] [d00000000c020ec0] cache_defer_hash+0x3258/0xffffffffffff0d68 [sunrpc] (unreliable) 2012-11-19 15:15:50 [c000000f498c7cd0] [d00000000bfef154] .__rpc_execute+0xc4/0x470 [sunrpc] 2012-11-19 15:15:50 [c000000f498c7db0] [c0000000000b6cac] .worker_thread+0x1dc/0x3c0 2012-11-19 15:15:50 [c000000f498c7ed0] [c0000000000bd02c] .kthread+0xbc/0xd0 2012-11-19 15:15:50 [c000000f498c7f90] [c000000000032054] .kernel_thread+0x54/0x70 2012-11-19 15:15:50 Mem-Info: 2012-11-19 15:15:50 Node 0 DMA per-cpu: 2012-11-19 15:15:50 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 2: hi: 6, btch: 1 usd: 4 2012-11-19 15:15:50 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 4: hi: 6, btch: 1 usd: 0 2012-11-19 15:15:50 CPU 5: hi: 6, btch: 1 usd: 0 2012-11-19 15:15:50 CPU 6: hi: 6, btch: 1 usd: 0 2012-11-19 15:15:50 CPU 7: hi: 6, btch: 1 usd: 2 2012-11-19 15:15:50 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 9: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 10: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 11: hi: 6, btch: 1 usd: 4 2012-11-19 15:15:50 CPU 12: hi: 6, btch: 1 usd: 4 2012-11-19 15:15:50 CPU 13: hi: 6, btch: 1 usd: 0 2012-11-19 15:15:50 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 18: hi: 6, btch: 1 usd: 0 2012-11-19 15:15:50 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 24: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 25: hi: 6, btch: 1 usd: 2 2012-11-19 15:15:50 CPU 26: hi: 6, btch: 1 usd: 2 2012-11-19 15:15:50 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 15:15:50 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 15:15:50 CPU 29: hi: 6, btch: 1 usd: 1 2012-11-19 15:15:50 CPU 30: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 31: hi: 6, btch: 1 usd: 3 2012-11-19 15:15:50 CPU 32: hi: 6, btch: 1 usd: 3 2012-11-19 15:15:50 CPU 33: hi: 6, btch: 1 usd: 1 2012-11-19 15:15:50 CPU 34: hi: 6, btch: 1 usd: 1 2012-11-19 15:15:50 CPU 35: hi: 6, btch: 1 usd: 0 2012-11-19 15:15:50 CPU 36: hi: 6, btch: 1 usd: 2 2012-11-19 15:15:50 CPU 37: hi: 6, btch: 1 usd: 0 2012-11-19 15:15:50 CPU 38: hi: 6, btch: 1 usd: 4 2012-11-19 15:15:50 CPU 39: hi: 6, btch: 1 usd: 1 2012-11-19 15:15:50 CPU 40: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 15:15:50 CPU 42: hi: 6, btch: 1 usd: 3 2012-11-19 15:15:50 CPU 43: hi: 6, btch: 1 usd: 3 2012-11-19 15:15:50 CPU 44: hi: 6, btch: 1 usd: 0 2012-11-19 15:15:50 CPU 45: hi: 6, btch: 1 usd: 0 2012-11-19 15:15:50 CPU 46: hi: 6, btch: 1 usd: 0 2012-11-19 15:15:50 CPU 47: hi: 6, btch: 1 usd: 0 2012-11-19 15:15:50 active_anon:6065 inactive_anon:4956 isolated_anon:0 2012-11-19 15:15:50 active_file:13101 inactive_file:922636 isolated_file:0 2012-11-19 15:15:50 unevictable:0 dirty:56 writeback:0 unstable:0 2012-11-19 15:15:50 free:596 slab_reclaimable:4671 slab_unreclaimable:20445 2012-11-19 15:15:50 mapped:1591 shmem:82 pagetables:650 bounce:0 2012-11-19 15:15:50 Node 0 DMA free:37952kB min:32128kB low:40128kB high:48192kB active_anon:388160kB inactive_anon:317184kB active_file:838464kB inactive_file:59048704kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:3584kB writeback:0kB mapped:101824kB shmem:5248kB slab_reclaimable:298944kB slab_unreclaimable:1308480kB kernel_stack:27184kB pagetables:41600kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 15:15:50 lowmem_reserve[]: 0 0 0 2012-11-19 15:15:50 Node 0 DMA: 520*64kB 11*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 37760kB 2012-11-19 15:15:50 936002 total pagecache pages 2012-11-19 15:15:50 0 pages in swap cache 2012-11-19 15:15:50 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 15:15:50 Free swap = 4194176kB 2012-11-19 15:15:50 Total swap = 4194176kB 2012-11-19 15:15:50 1011712 pages RAM 2012-11-19 15:15:50 20742 pages reserved 2012-11-19 15:15:50 31046 pages shared 2012-11-19 15:15:50 970403 pages non-shared 2012-11-19 15:19:08 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 15:19:08 Call Trace: 2012-11-19 15:19:08 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 15:19:08 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 15:19:08 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 15:19:08 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 15:19:08 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 15:19:08 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 15:19:08 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 15:19:08 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 15:19:08 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 15:19:08 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 15:19:08 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 15:19:08 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 15:19:08 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 15:19:08 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 15:19:08 [c000000f59b7b970] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 15:19:08 [c000000f59b7ba10] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 15:19:08 [c000000f59b7ba90] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 15:19:08 [c000000f59b7bb40] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 15:19:08 --- Exception: 501 at .raw_local_irq_restore+0x34/0x60 2012-11-19 15:19:08 LR = .cpu_idle+0x14c/0x1d0 2012-11-19 15:19:08 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 (unreliable) 2012-11-19 15:19:08 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 15:19:08 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 15:19:08 Mem-Info: 2012-11-19 15:19:08 Node 0 DMA per-cpu: 2012-11-19 15:19:08 CPU 0: hi: 6, btch: 1 usd: 4 2012-11-19 15:19:08 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 5: hi: 6, btch: 1 usd: 1 2012-11-19 15:19:08 CPU 6: hi: 6, btch: 1 usd: 4 2012-11-19 15:19:08 CPU 7: hi: 6, btch: 1 usd: 1 2012-11-19 15:19:08 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 15:19:08 CPU 10: hi: 6, btch: 1 usd: 2 2012-11-19 15:19:08 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 18: hi: 6, btch: 1 usd: 1 2012-11-19 15:19:08 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 20: hi: 6, btch: 1 usd: 2 2012-11-19 15:19:08 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 24: hi: 6, btch: 1 usd: 0 2012-11-19 15:19:08 CPU 25: hi: 6, btch: 1 usd: 2 2012-11-19 15:19:08 CPU 26: hi: 6, btch: 1 usd: 1 2012-11-19 15:19:08 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 15:19:08 CPU 28: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 29: hi: 6, btch: 1 usd: 0 2012-11-19 15:19:08 CPU 30: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 15:19:08 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 33: hi: 6, btch: 1 usd: 3 2012-11-19 15:19:08 CPU 34: hi: 6, btch: 1 usd: 1 2012-11-19 15:19:08 CPU 35: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 36: hi: 6, btch: 1 usd: 2 2012-11-19 15:19:08 CPU 37: hi: 6, btch: 1 usd: 2 2012-11-19 15:19:08 CPU 38: hi: 6, btch: 1 usd: 3 2012-11-19 15:19:08 CPU 39: hi: 6, btch: 1 usd: 3 2012-11-19 15:19:08 CPU 40: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 42: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 15:19:08 CPU 44: hi: 6, btch: 1 usd: 4 2012-11-19 15:19:08 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 15:19:08 CPU 47: hi: 6, btch: 1 usd: 3 2012-11-19 15:19:08 active_anon:5809 inactive_anon:4956 isolated_anon:0 2012-11-19 15:19:08 active_file:13722 inactive_file:918839 isolated_file:0 2012-11-19 15:19:08 unevictable:0 dirty:271 writeback:50 unstable:0 2012-11-19 15:19:08 free:747 slab_reclaimable:5866 slab_unreclaimable:22300 2012-11-19 15:19:08 mapped:1643 shmem:82 pagetables:641 bounce:0 2012-11-19 15:19:08 Node 0 DMA free:47616kB min:32128kB low:40128kB high:48192kB active_anon:371776kB inactive_anon:317184kB active_file:878208kB inactive_file:58805696kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:17344kB writeback:3200kB mapped:105152kB shmem:5248kB slab_reclaimable:375424kB slab_unreclaimable:1427200kB kernel_stack:27184kB pagetables:41024kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 15:19:08 lowmem_reserve[]: 0 0 0 2012-11-19 15:19:08 Node 0 DMA: 642*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 44288kB 2012-11-19 15:19:08 932106 total pagecache pages 2012-11-19 15:19:08 0 pages in swap cache 2012-11-19 15:19:08 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 15:19:08 Free swap = 4194176kB 2012-11-19 15:19:08 Total swap = 4194176kB 2012-11-19 15:19:08 1011712 pages RAM 2012-11-19 15:19:08 20742 pages reserved 2012-11-19 15:19:08 20765 pages shared 2012-11-19 15:19:08 979050 pages non-shared 2012-11-19 15:27:31 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 15:27:31 Call Trace: 2012-11-19 15:27:31 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 15:27:31 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 15:27:31 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 15:27:31 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 15:27:31 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 15:27:31 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 15:27:31 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 15:27:31 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 15:27:31 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 15:27:31 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 15:27:31 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 15:27:31 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 15:27:31 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 15:27:31 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 15:27:31 [c000000f59b7b970] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 15:27:31 [c000000f59b7ba10] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 15:27:31 [c000000f59b7ba90] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 15:27:31 [c000000f59b7bb40] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 15:27:31 --- Exception: 501 at .raw_local_irq_restore+0x34/0x60 2012-11-19 15:27:31 LR = .cpu_idle+0x14c/0x1d0 2012-11-19 15:27:31 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 (unreliable) 2012-11-19 15:27:31 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 15:27:31 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 15:27:31 Mem-Info: 2012-11-19 15:27:31 Node 0 DMA per-cpu: 2012-11-19 15:27:31 CPU 0: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 15:27:31 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 15:27:31 CPU 3: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 15:27:31 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 15:27:31 CPU 6: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 7: hi: 6, btch: 1 usd: 4 2012-11-19 15:27:31 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 15:27:31 CPU 9: hi: 6, btch: 1 usd: 4 2012-11-19 15:27:31 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 11: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 12: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 13: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 14: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 15: hi: 6, btch: 1 usd: 4 2012-11-19 15:27:31 CPU 16: hi: 6, btch: 1 usd: 3 2012-11-19 15:27:31 CPU 17: hi: 6, btch: 1 usd: 4 2012-11-19 15:27:31 CPU 18: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 19: hi: 6, btch: 1 usd: 4 2012-11-19 15:27:31 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 15:27:31 CPU 21: hi: 6, btch: 1 usd: 1 2012-11-19 15:27:31 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 15:27:31 CPU 23: hi: 6, btch: 1 usd: 3 2012-11-19 15:27:31 CPU 24: hi: 6, btch: 1 usd: 4 2012-11-19 15:27:31 CPU 25: hi: 6, btch: 1 usd: 2 2012-11-19 15:27:31 CPU 26: hi: 6, btch: 1 usd: 4 2012-11-19 15:27:31 CPU 27: hi: 6, btch: 1 usd: 2 2012-11-19 15:27:31 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 29: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 30: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 15:27:31 CPU 33: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 35: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 36: hi: 6, btch: 1 usd: 5 2012-11-19 15:27:31 CPU 37: hi: 6, btch: 1 usd: 5 2012-11-19 15:27:31 CPU 38: hi: 6, btch: 1 usd: 5 2012-11-19 15:27:31 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 40: hi: 6, btch: 1 usd: 5 2012-11-19 15:27:31 CPU 41: hi: 6, btch: 1 usd: 1 2012-11-19 15:27:31 CPU 42: hi: 6, btch: 1 usd: 4 2012-11-19 15:27:31 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 44: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 45: hi: 6, btch: 1 usd: 4 2012-11-19 15:27:31 CPU 46: hi: 6, btch: 1 usd: 0 2012-11-19 15:27:31 CPU 47: hi: 6, btch: 1 usd: 1 2012-11-19 15:27:31 active_anon:5037 inactive_anon:4955 isolated_anon:0 2012-11-19 15:27:31 active_file:32551 inactive_file:886053 isolated_file:32 2012-11-19 15:27:31 unevictable:0 dirty:268 writeback:0 unstable:0 2012-11-19 15:27:31 free:1760 slab_reclaimable:8162 slab_unreclaimable:34096 2012-11-19 15:27:31 mapped:1616 shmem:83 pagetables:658 bounce:0 2012-11-19 15:27:31 Node 0 DMA free:112640kB min:32128kB low:40128kB high:48192kB active_anon:322368kB inactive_anon:317120kB active_file:2083264kB inactive_file:56707392kB unevictable:0kB isolated(anon):0kB isolated(file):2048kB present:64694208kB mlocked:0kB dirty:17152kB writeback:0kB mapped:103424kB shmem:5312kB slab_reclaimable:522368kB slab_unreclaimable:2182144kB kernel_stack:27296kB pagetables:42112kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 15:27:31 lowmem_reserve[]: 0 0 0 2012-11-19 15:27:31 Node 0 DMA: 1251*64kB 63*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 88128kB 2012-11-19 15:27:31 918783 total pagecache pages 2012-11-19 15:27:31 0 pages in swap cache 2012-11-19 15:27:31 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 15:27:31 Free swap = 4194176kB 2012-11-19 15:27:31 Total swap = 4194176kB 2012-11-19 15:27:31 1011712 pages RAM 2012-11-19 15:27:31 20742 pages reserved 2012-11-19 15:27:31 61956 pages shared 2012-11-19 15:27:31 941621 pages non-shared 2012-11-19 15:28:14 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 15:28:14 Call Trace: 2012-11-19 15:28:14 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 15:28:14 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 15:28:14 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 15:28:14 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 15:28:14 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 15:28:14 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 15:28:14 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 15:28:14 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 15:28:14 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 15:28:14 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 15:28:14 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 15:28:14 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 15:28:14 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 15:28:14 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 15:28:14 [c000000f59b7b970] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 15:28:14 [c000000f59b7ba10] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 15:28:14 [c000000f59b7ba90] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 15:28:14 [c000000f59b7bb40] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 15:28:14 --- Exception: 501 at .raw_local_irq_restore+0x34/0x60 2012-11-19 15:28:14 LR = .cpu_idle+0x14c/0x1d0 2012-11-19 15:28:14 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 (unreliable) 2012-11-19 15:28:14 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 15:28:14 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 15:28:14 Mem-Info: 2012-11-19 15:28:14 Node 0 DMA per-cpu: 2012-11-19 15:28:14 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 3: hi: 6, btch: 1 usd: 1 2012-11-19 15:28:14 CPU 4: hi: 6, btch: 1 usd: 1 2012-11-19 15:28:14 CPU 5: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 6: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 7: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 10: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 11: hi: 6, btch: 1 usd: 2 2012-11-19 15:28:14 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 16: hi: 6, btch: 1 usd: 2 2012-11-19 15:28:14 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 24: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 25: hi: 6, btch: 1 usd: 3 2012-11-19 15:28:14 CPU 26: hi: 6, btch: 1 usd: 4 2012-11-19 15:28:14 CPU 27: hi: 6, btch: 1 usd: 3 2012-11-19 15:28:14 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 29: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 30: hi: 6, btch: 1 usd: 3 2012-11-19 15:28:14 CPU 31: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 33: hi: 6, btch: 1 usd: 2 2012-11-19 15:28:14 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 35: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 36: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 37: hi: 6, btch: 1 usd: 4 2012-11-19 15:28:14 CPU 38: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 39: hi: 6, btch: 1 usd: 2 2012-11-19 15:28:14 CPU 40: hi: 6, btch: 1 usd: 2 2012-11-19 15:28:14 CPU 41: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 42: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 43: hi: 6, btch: 1 usd: 0 2012-11-19 15:28:14 CPU 44: hi: 6, btch: 1 usd: 1 2012-11-19 15:28:14 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 15:28:14 CPU 47: hi: 6, btch: 1 usd: 3 2012-11-19 15:28:14 active_anon:4976 inactive_anon:4955 isolated_anon:0 2012-11-19 15:28:14 active_file:32475 inactive_file:887500 isolated_file:32 2012-11-19 15:28:14 unevictable:0 dirty:619 writeback:0 unstable:0 2012-11-19 15:28:14 free:835 slab_reclaimable:8121 slab_unreclaimable:33614 2012-11-19 15:28:14 mapped:1570 shmem:83 pagetables:655 bounce:0 2012-11-19 15:28:14 Node 0 DMA free:53824kB min:32128kB low:40128kB high:48192kB active_anon:318464kB inactive_anon:317120kB active_file:2078400kB inactive_file:56800000kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:39616kB writeback:0kB mapped:100480kB shmem:5312kB slab_reclaimable:519744kB slab_unreclaimable:2151296kB kernel_stack:27216kB pagetables:41920kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 15:28:14 lowmem_reserve[]: 0 0 0 2012-11-19 15:28:14 Node 0 DMA: 367*64kB 54*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 30400kB 2012-11-19 15:28:14 920289 total pagecache pages 2012-11-19 15:28:14 0 pages in swap cache 2012-11-19 15:28:14 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 15:28:14 Free swap = 4194176kB 2012-11-19 15:28:14 Total swap = 4194176kB 2012-11-19 15:28:14 1011712 pages RAM 2012-11-19 15:28:14 20742 pages reserved 2012-11-19 15:28:14 71102 pages shared 2012-11-19 15:28:14 932975 pages non-shared 2012-11-19 15:40:40 rpciod/8: page allocation failure. order:1, mode:0x20 2012-11-19 15:40:40 Call Trace: 2012-11-19 15:40:40 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 15:40:40 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 15:40:40 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 15:40:40 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 15:40:40 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 15:40:40 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 15:40:40 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 15:40:40 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 15:40:40 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 15:40:40 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 15:40:40 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 15:40:40 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 15:40:40 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 15:40:40 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 15:40:40 [c000000f498c6e20] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 15:40:40 [c000000f498c6ec0] [c00000000009c92c] .local_bh_enable+0xbc/0xd0 2012-11-19 15:40:40 [c000000f498c6f40] [c0000000004e6a6c] .dev_queue_xmit+0x25c/0x8e0 2012-11-19 15:40:40 [c000000f498c6ff0] [c0000000004efc70] .neigh_connected_output+0x110/0x160 2012-11-19 15:40:40 [c000000f498c70a0] [c00000000053079c] .ip_finish_output+0x6c/0x3e0 2012-11-19 15:40:40 [c000000f498c7140] [c000000000530fa4] .ip_local_out+0x44/0x60 2012-11-19 15:40:40 [c000000f498c71c0] [c0000000005314fc] .ip_queue_xmit+0x16c/0x410 2012-11-19 15:40:40 [c000000f498c72c0] [c000000000549fd4] .tcp_transmit_skb+0x474/0x9b0 2012-11-19 15:40:40 [c000000f498c73c0] [c00000000054d020] .tcp_write_xmit+0x230/0xbb0 2012-11-19 15:40:40 [c000000f498c74d0] [c00000000054da20] .__tcp_push_pending_frames+0x40/0x100 2012-11-19 15:40:40 [c000000f498c7550] [c00000000053d228] .tcp_sendmsg+0x2a8/0xdf0 2012-11-19 15:40:40 [c000000f498c76a0] [c0000000004ca3ec] .sock_sendmsg+0x14c/0x1a0 2012-11-19 15:40:40 [c000000f498c78c0] [c0000000004ca478] .kernel_sendmsg+0x38/0x60 2012-11-19 15:40:40 [c000000f498c7940] [d00000000bfe8ac4] .xs_send_kvec+0xb4/0xd0 [sunrpc] 2012-11-19 15:40:41 [c000000f498c7a00] [d00000000bfe8b9c] .xs_sendpages+0xbc/0x350 [sunrpc] 2012-11-19 15:40:41 [c000000f498c7ad0] [d00000000bfe8fdc] .xs_tcp_send_request+0x6c/0x1e0 [sunrpc] 2012-11-19 15:40:41 [c000000f498c7b80] [d00000000bfe59a4] .xprt_transmit+0xa4/0x340 [sunrpc] 2012-11-19 15:40:41 [c000000f498c7c40] [d00000000bfe1928] .call_transmit+0x1e8/0x2c0 [sunrpc] 2012-11-19 15:40:41 [c000000f498c7cd0] [d00000000bfef154] .__rpc_execute+0xc4/0x470 [sunrpc] 2012-11-19 15:40:41 [c000000f498c7db0] [c0000000000b6cac] .worker_thread+0x1dc/0x3c0 2012-11-19 15:40:41 [c000000f498c7ed0] [c0000000000bd02c] .kthread+0xbc/0xd0 2012-11-19 15:40:41 [c000000f498c7f90] [c000000000032054] .kernel_thread+0x54/0x70 2012-11-19 15:40:41 Mem-Info: 2012-11-19 15:40:41 Node 0 DMA per-cpu: 2012-11-19 15:40:41 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 2: hi: 6, btch: 1 usd: 4 2012-11-19 15:40:41 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 8: hi: 6, btch: 1 usd: 3 2012-11-19 15:40:41 CPU 9: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 10: hi: 6, btch: 1 usd: 3 2012-11-19 15:40:41 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 24: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 25: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 26: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 27: hi: 6, btch: 1 usd: 0 2012-11-19 15:40:41 CPU 28: hi: 6, btch: 1 usd: 0 2012-11-19 15:40:41 CPU 29: hi: 6, btch: 1 usd: 2 2012-11-19 15:40:41 CPU 30: hi: 6, btch: 1 usd: 0 2012-11-19 15:40:41 CPU 31: hi: 6, btch: 1 usd: 3 2012-11-19 15:40:41 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 33: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 34: hi: 6, btch: 1 usd: 0 2012-11-19 15:40:41 CPU 35: hi: 6, btch: 1 usd: 3 2012-11-19 15:40:41 CPU 36: hi: 6, btch: 1 usd: 3 2012-11-19 15:40:41 CPU 37: hi: 6, btch: 1 usd: 2 2012-11-19 15:40:41 CPU 38: hi: 6, btch: 1 usd: 4 2012-11-19 15:40:41 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 15:40:41 CPU 40: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 41: hi: 6, btch: 1 usd: 3 2012-11-19 15:40:41 CPU 42: hi: 6, btch: 1 usd: 3 2012-11-19 15:40:41 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 44: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 15:40:41 active_anon:5416 inactive_anon:4957 isolated_anon:0 2012-11-19 15:40:41 active_file:24154 inactive_file:901759 isolated_file:0 2012-11-19 15:40:41 unevictable:0 dirty:215 writeback:0 unstable:0 2012-11-19 15:40:41 free:2093 slab_reclaimable:6914 slab_unreclaimable:27239 2012-11-19 15:40:41 mapped:1612 shmem:84 pagetables:676 bounce:0 2012-11-19 15:40:41 Node 0 DMA free:133952kB min:32128kB low:40128kB high:48192kB active_anon:346624kB inactive_anon:317248kB active_file:1545856kB inactive_file:57712576kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:13760kB writeback:0kB mapped:103168kB shmem:5376kB slab_reclaimable:442496kB slab_unreclaimable:1743296kB kernel_stack:27376kB pagetables:43264kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 15:40:41 lowmem_reserve[]: 0 0 0 2012-11-19 15:40:41 Node 0 DMA: 1773*64kB 139*128kB 3*256kB 7*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 135616kB 2012-11-19 15:40:41 924857 total pagecache pages 2012-11-19 15:40:41 0 pages in swap cache 2012-11-19 15:40:41 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 15:40:41 Free swap = 4194176kB 2012-11-19 15:40:41 Total swap = 4194176kB 2012-11-19 15:40:41 1011712 pages RAM 2012-11-19 15:40:41 20742 pages reserved 2012-11-19 15:40:41 22168 pages shared 2012-11-19 15:40:41 977894 pages non-shared Console [seqlac2] log at 2012-11-19 16:00:00 PST. 2012-11-19 16:03:44 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 16:03:44 Call Trace: 2012-11-19 16:03:44 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 16:03:44 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 16:03:44 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 16:03:44 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 16:03:44 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 16:03:44 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 16:03:44 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 16:03:44 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 16:03:44 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 16:03:44 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 16:03:44 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 16:03:44 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 16:03:44 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 16:03:44 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 16:03:44 [c000000f59b7b970] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 16:03:44 [c000000f59b7ba10] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 16:03:44 [c000000f59b7ba90] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 16:03:44 --- Exception: 501 at 0x3d226e32594c 2012-11-19 16:03:44 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 16:03:44 [c000000f59b7bb40] [c000000000004898] hardware_interrupt_entry+0x18/0x80 (unreliable) 2012-11-19 16:03:44 --- Exception: 501 at .raw_local_irq_restore+0x34/0x60 2012-11-19 16:03:44 LR = .cpu_idle+0x14c/0x1d0 2012-11-19 16:03:44 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 (unreliable) 2012-11-19 16:03:44 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 16:03:44 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 16:03:44 Mem-Info: 2012-11-19 16:03:44 Node 0 DMA per-cpu: 2012-11-19 16:03:44 CPU 0: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 1: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 2: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 5: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 9: hi: 6, btch: 1 usd: 0 2012-11-19 16:03:44 CPU 10: hi: 6, btch: 1 usd: 3 2012-11-19 16:03:44 CPU 11: hi: 6, btch: 1 usd: 4 2012-11-19 16:03:44 CPU 12: hi: 6, btch: 1 usd: 0 2012-11-19 16:03:44 CPU 13: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 24: hi: 6, btch: 1 usd: 4 2012-11-19 16:03:44 CPU 25: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 26: hi: 6, btch: 1 usd: 1 2012-11-19 16:03:44 CPU 27: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 28: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 29: hi: 6, btch: 1 usd: 0 2012-11-19 16:03:44 CPU 30: hi: 6, btch: 1 usd: 0 2012-11-19 16:03:44 CPU 31: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 32: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 33: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 34: hi: 6, btch: 1 usd: 4 2012-11-19 16:03:44 CPU 35: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 36: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 37: hi: 6, btch: 1 usd: 1 2012-11-19 16:03:44 CPU 38: hi: 6, btch: 1 usd: 1 2012-11-19 16:03:44 CPU 39: hi: 6, btch: 1 usd: 0 2012-11-19 16:03:44 CPU 40: hi: 6, btch: 1 usd: 4 2012-11-19 16:03:44 CPU 41: hi: 6, btch: 1 usd: 4 2012-11-19 16:03:44 CPU 42: hi: 6, btch: 1 usd: 0 2012-11-19 16:03:44 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 44: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 16:03:44 CPU 46: hi: 6, btch: 1 usd: 4 2012-11-19 16:03:44 CPU 47: hi: 6, btch: 1 usd: 0 2012-11-19 16:03:44 active_anon:5542 inactive_anon:4955 isolated_anon:0 2012-11-19 16:03:44 active_file:12852 inactive_file:919875 isolated_file:0 2012-11-19 16:03:44 unevictable:0 dirty:295 writeback:48 unstable:0 2012-11-19 16:03:44 free:755 slab_reclaimable:5267 slab_unreclaimable:23214 2012-11-19 16:03:44 mapped:1694 shmem:82 pagetables:687 bounce:0 2012-11-19 16:03:44 Node 0 DMA free:48320kB min:32128kB low:40128kB high:48192kB active_anon:354688kB inactive_anon:317120kB active_file:822528kB inactive_file:58872000kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:64694208kB mlocked:0kB dirty:18880kB writeback:3072kB mapped:108416kB shmem:5248kB slab_reclaimable:337088kB slab_unreclaimable:1485696kB kernel_stack:27408kB pagetables:43968kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 16:03:44 lowmem_reserve[]: 0 0 0 2012-11-19 16:03:44 Node 0 DMA: 759*64kB 38*128kB 10*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 58048kB 2012-11-19 16:03:44 931351 total pagecache pages 2012-11-19 16:03:44 0 pages in swap cache 2012-11-19 16:03:44 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 16:03:44 Free swap = 4194176kB 2012-11-19 16:03:44 Total swap = 4194176kB 2012-11-19 16:03:44 1011712 pages RAM 2012-11-19 16:03:44 20742 pages reserved 2012-11-19 16:03:44 23124 pages shared 2012-11-19 16:03:44 978177 pages non-shared 2012-11-19 16:17:44 swapper: page allocation failure. order:1, mode:0x20 2012-11-19 16:17:44 Call Trace: 2012-11-19 16:17:44 [c000000f6ff8f510] [c000000000012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 16:17:44 [c000000f6ff8f5c0] [c00000000016b898] .__alloc_pages_nodemask+0x618/0x930 2012-11-19 16:17:44 [c000000f6ff8f740] [c0000000001b118c] .kmem_getpages+0x7c/0x1a0 2012-11-19 16:17:44 [c000000f6ff8f7d0] [c0000000001b2594] .fallback_alloc+0x244/0x310 2012-11-19 16:17:44 [c000000f6ff8f8b0] [c0000000001b26f0] .kmem_cache_alloc_node_notrace+0x90/0x1c0 2012-11-19 16:17:44 [c000000f6ff8f960] [c0000000001b29f4] .__kmalloc_node+0x94/0x140 2012-11-19 16:17:44 [c000000f6ff8fa20] [c0000000004d79cc] .__alloc_skb+0x7c/0x1b0 2012-11-19 16:17:44 [c000000f6ff8fac0] [c0000000004d8af4] .dev_alloc_skb+0x24/0x60 2012-11-19 16:17:44 [c000000f6ff8fb30] [d00000000b9ea9bc] .ipoib_cm_alloc_rx_skb+0x5c/0x550 [ib_ipoib] 2012-11-19 16:17:44 [c000000f6ff8fc00] [d00000000b9ef400] .ipoib_cm_handle_rx_wc+0x2c0/0x740 [ib_ipoib] 2012-11-19 16:17:44 [c000000f6ff8fcf0] [d00000000b9e64e8] .ipoib_poll+0x138/0x230 [ib_ipoib] 2012-11-19 16:17:44 [c000000f6ff8fdc0] [c0000000004e5a88] .net_rx_action+0x178/0x450 2012-11-19 16:17:44 [c000000f6ff8feb0] [c00000000009cb48] .__do_softirq+0x118/0x290 2012-11-19 16:17:44 [c000000f6ff8ff90] [c000000000031e28] .call_do_softirq+0x14/0x24 2012-11-19 16:17:44 [c000000f59b7b8c0] [c00000000000e700] .do_softirq+0xf0/0x110 2012-11-19 16:17:44 [c000000f59b7b960] [c00000000009c864] .irq_exit+0xb4/0xc0 2012-11-19 16:17:44 [c000000f59b7b9e0] [c00000000000e964] .do_IRQ+0x144/0x230 2012-11-19 16:17:44 [c000000f59b7ba90] [c000000000004898] hardware_interrupt_entry+0x18/0x80 2012-11-19 16:17:44 --- Exception: 501 at .ppc64_runlatch_off+0x24/0x60 2012-11-19 16:17:44 LR = .pseries_dedicated_idle_sleep+0x10c/0x1f0 2012-11-19 16:17:44 [c000000f59b7bd80] [0001eae978eac7d9] 0x1eae978eac7d9 (unreliable) 2012-11-19 16:17:44 [c000000f59b7be30] [c000000000015bf0] .cpu_idle+0x140/0x1d0 2012-11-19 16:17:44 [c000000f59b7bee0] [c0000000005c9b44] .start_secondary+0x34c/0x384 2012-11-19 16:17:44 [c000000f59b7bf90] [c000000000008268] .start_secondary_prolog+0x10/0x14 2012-11-19 16:17:44 Mem-Info: 2012-11-19 16:17:44 Node 0 DMA per-cpu: 2012-11-19 16:17:44 CPU 0: hi: 6, btch: 1 usd: 0 2012-11-19 16:17:44 CPU 1: hi: 6, btch: 1 usd: 0 2012-11-19 16:17:44 CPU 2: hi: 6, btch: 1 usd: 2 2012-11-19 16:17:44 CPU 3: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 4: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 5: hi: 6, btch: 1 usd: 0 2012-11-19 16:17:44 CPU 6: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 7: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 8: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 9: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 10: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 11: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 12: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 13: hi: 6, btch: 1 usd: 1 2012-11-19 16:17:44 CPU 14: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 15: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 16: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 17: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 18: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 19: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 20: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 21: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 22: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 23: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 24: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 25: hi: 6, btch: 1 usd: 0 2012-11-19 16:17:44 CPU 26: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 27: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 28: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 29: hi: 6, btch: 1 usd: 0 2012-11-19 16:17:44 CPU 30: hi: 6, btch: 1 usd: 0 2012-11-19 16:17:44 CPU 31: hi: 6, btch: 1 usd: 4 2012-11-19 16:17:44 CPU 32: hi: 6, btch: 1 usd: 0 2012-11-19 16:17:44 CPU 33: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 34: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 35: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 36: hi: 6, btch: 1 usd: 3 2012-11-19 16:17:44 CPU 37: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 38: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 39: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 40: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 41: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 42: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 43: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 44: hi: 6, btch: 1 usd: 4 2012-11-19 16:17:44 CPU 45: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 46: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 CPU 47: hi: 6, btch: 1 usd: 5 2012-11-19 16:17:44 active_anon:4973 inactive_anon:4800 isolated_anon:0 2012-11-19 16:17:44 active_file:23087 inactive_file:912991 isolated_file:32 2012-11-19 16:17:44 unevictable:0 dirty:506 writeback:52 unstable:0 2012-11-19 16:17:44 free:1783 slab_reclaimable:4570 slab_unreclaimable:20399 2012-11-19 16:17:44 mapped:1561 shmem:81 pagetables:695 bounce:0 2012-11-19 16:17:44 Node 0 DMA free:115072kB min:32128kB low:40128kB high:48192kB active_anon:318272kB inactive_anon:307200kB active_file:1477568kB inactive_file:58431424kB unevictable:0kB isolated(anon):0kB isolated(file):2048kB present:64694208kB mlocked:0kB dirty:32384kB writeback:3328kB mapped:99904kB shmem:5184kB slab_reclaimable:292480kB slab_unreclaimable:1305536kB kernel_stack:28112kB pagetables:44480kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2012-11-19 16:17:44 lowmem_reserve[]: 0 0 0 2012-11-19 16:17:44 Node 0 DMA: 1542*64kB 117*128kB 7*256kB 8*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 119552kB 2012-11-19 16:17:44 936167 total pagecache pages 2012-11-19 16:17:44 0 pages in swap cache 2012-11-19 16:17:44 Swap cache stats: add 0, delete 0, find 0/0 2012-11-19 16:17:44 Free swap = 4194176kB 2012-11-19 16:17:44 Total swap = 4194176kB 2012-11-19 16:17:44 1011712 pages RAM 2012-11-19 16:17:44 20742 pages reserved 2012-11-19 16:17:44 22151 pages shared 2012-11-19 16:17:44 977144 pages non-shared 2012-11-19 16:18:40 Unable to handle kernel paging request for data at address 0x00000008 2012-11-19 16:18:40 Faulting instruction address: 0xd0000000139b6710 2012-11-19 16:18:40 Oops: Kernel access of bad area, sig: 11 [#1] 2012-11-19 16:18:40 SMP NR_CPUS=1024 NUMA pSeries 2012-11-19 16:18:40 Modules linked in: xt_owner nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack lmv(U) mgc(U) lustre(U) mdc(U) fid(U) fld(U) lov(U) osc(U) ptlrpc(U) obdclass(U) lvfs(U) nfs fscache lockd nfs_acl auth_rpcgss ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) sunrpc ipt_LOG xt_multiport iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa uinput raid1 ses enclosure sg mlx4_ib ib_mad ib_core mlx4_en mlx4_core e1000e ehea ext4 jbd2 mbcache raid456 async_pq async_xor xor async_raid6_recov raid6_pq async_memcpy async_tx sd_mod crc_t10dif ipr dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] 2012-11-19 16:18:40 NIP: d0000000139b6710 LR: d0000000139b6700 CTR: c0000000005be960 2012-11-19 16:18:40 REGS: c000000f54aab080 TRAP: 0300 Not tainted (2.6.32-279.11.1.1chaos.bgq62.ppc64) 2012-11-19 16:18:40 MSR: 8000000000009032 CR: 24008442 XER: 20000000 2012-11-19 16:18:40 DAR: 0000000000000008, DSISR: 0000000042000000 2012-11-19 16:18:40 TASK = c000000f539312d0[10079] 'ls' THREAD: c000000f54aa8000 CPU: 32 2012-11-19 16:18:40 GPR00: c00000025f6e5d90 c000000f54aab300 d000000013a0d878 c000000ecf32d120 2012-11-19 16:18:40 GPR04: 5a5a5a5a5a5a5a5a 0000000000000004 c0000005ff4e506c 0000000000000002 2012-11-19 16:18:40 GPR08: c000000000ff7500 0000000000000000 c000000001099004 0000000000000000 2012-11-19 16:18:40 GPR12: d0000000139d2610 c000000000ff7500 0000000010022ff8 00000000000000e8 2012-11-19 16:18:40 GPR16: c000000739a8f200 0000000000000000 0000000000000001 0000000000000003 2012-11-19 16:18:40 GPR20: c000000ecf32cee0 0000000000000000 c00000025f6e5d80 0000000000000000 2012-11-19 16:18:40 GPR24: c000000f45ae0000 c00000025f6e5d80 d00000000cecfdc0 d00000000cecfdc4 2012-11-19 16:18:40 GPR28: 0000000000003300 c000000ecf32cd80 d000000013a0aca8 c000000f54aab300 2012-11-19 16:18:40 NIP [d0000000139b6710] .ll_sai_unplug+0x210/0x860 [lustre] 2012-11-19 16:18:40 LR [d0000000139b6700] .ll_sai_unplug+0x200/0x860 [lustre] 2012-11-19 16:18:40 Call Trace: 2012-11-19 16:18:40 [c000000f54aab300] [d0000000139b6700] .ll_sai_unplug+0x200/0x860 [lustre] (unreliable) 2012-11-19 16:18:40 [c000000f54aab430] [d0000000139b7bc8] .do_statahead_enter+0x218/0x22b0 [lustre] 2012-11-19 16:18:40 [c000000f54aab5f0] [d000000013933dec] .ll_revalidate_it+0x82c/0x21f0 [lustre] 2012-11-19 16:18:40 [c000000f54aab750] [d00000001393595c] .ll_revalidate_nd+0x1ac/0x560 [lustre] 2012-11-19 16:18:40 [c000000f54aab810] [c0000000001d55b0] .do_lookup+0xa0/0x2d0 2012-11-19 16:18:40 [c000000f54aab8e0] [c0000000001d8c50] .__link_path_walk+0x9b0/0x15a0 2012-11-19 16:18:40 [c000000f54aaba10] [c0000000001d9bf8] .path_walk+0x98/0x180 2012-11-19 16:18:40 [c000000f54aabab0] [c0000000001d9eec] .do_path_lookup+0x7c/0xf0 2012-11-19 16:18:40 [c000000f54aabb40] [c0000000001dac50] .user_path_at+0x60/0xb0 2012-11-19 16:18:40 [c000000f54aabc90] [c0000000001ce144] .vfs_fstatat+0x44/0xb0 2012-11-19 16:18:40 [c000000f54aabd30] [c0000000001ce314] .SyS_newlstat+0x24/0x50 2012-11-19 16:18:40 [c000000f54aabe30] [c000000000008564] syscall_exit+0x0/0x40 2012-11-19 16:18:40 Instruction dump: 2012-11-19 16:18:40 81f90058 ea1d0000 79ef06e0 3a10ff08 39ef00e8 79e31764 7c7d1a14 4801bf15 2012-11-19 16:18:40 e8410028 e9390018 e9790010 38190010 f9690000 f8190018 f8190010 2012-11-19 16:18:40 Sending IPI to other cpus... 2012-11-19 16:19:28 bootmem alloc of 8388608 bytes failed! 2012-11-19 16:19:28 Kernel panic - not syncing: Out of memory 2012-11-19 16:19:28 Call Trace: 2012-11-19 16:19:28 [c000000004f27bc0] [c000000004012f04] .show_stack+0x74/0x1c0 (unreliable) 2012-11-19 16:19:28 [c000000004f27c70] [c0000000045c4128] .panic+0xb8/0x1ec 2012-11-19 16:19:28 [c000000004f27d00] [c00000000486e94c] .___alloc_bootmem+0x58/0x5c 2012-11-19 16:19:28 [c000000004f27d80] [c000000004872138] .sparse_init+0x54/0x304 2012-11-19 16:19:28 [c000000004f27e50] [c0000000048561dc] .setup_arch+0x204/0x290 2012-11-19 16:19:28 [c000000004f27ef0] [c0000000048509f4] .start_kernel+0xb0/0x590 2012-11-19 16:19:28 [c000000004f27f90] [c0000000040083d8] .start_here_common+0x1c/0x44 2012-11-19 16:19:28 Rebooting in 180 seconds..