Aug  9 03:10:01 cs04r-sc-mds03-01 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3852" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Aug  9 11:05:30 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000-osd: FID [0x200010bf1:0x1aa56:0x0] != self_fid [0x200010bf1:0x1b258:0x0]
Aug  9 17:37:59 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000-osd: FID [0x200010c4b:0x1ddcf:0x0] != self_fid [0x200010c4c:0xfc3:0x0]
Aug  9 17:41:40 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000-osd: FID [0x200010c4c:0x1a6d6:0x0] != self_fid [0x200010c4c:0x1da8b:0x0]
Aug  9 17:51:16 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000-osd: FID [0x200010c4f:0x67eb:0x0] != self_fid [0x200010c4f:0xb08a:0x0]
Aug  9 18:16:39 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000-osd: FID [0x200010c55:0x180b7:0x0] != self_fid [0x200010c55:0x1bf40:0x0]
Aug  9 18:41:14 cs04r-sc-mds03-01 kernel: Lustre: MGS: haven't heard from client 7803d73f-6185-32d8-1a73-528f2eef485a (at 172.23.140.3@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881a53445800, cur 1439142074 expire 1439141924 last 1439141847
Aug  9 18:41:17 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000: haven't heard from client dbdc91dd-40a8-3b9e-34b6-a5508e072323 (at 10.144.140.3@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805ac552000, cur 1439142077 expire 1439141927 last 1439141850
Aug  9 19:02:54 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000: haven't heard from client f2a6d926-5e09-7040-3b78-55aa9093cca3 (at 10.144.152.28@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fc55aac00, cur 1439143374 expire 1439143224 last 1439143147
Aug  9 19:12:30 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000: haven't heard from client eb07be4f-95ad-a590-4786-44ee81c6aef2 (at 10.144.152.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff9036800, cur 1439143950 expire 1439143800 last 1439143723
Aug  9 19:12:30 cs04r-sc-mds03-01 kernel: Lustre: Skipped 1 previous similar message
Aug 10 08:54:49 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000: haven't heard from client 60240115-81a1-d48b-6ea1-36dabacaa090 (at 10.144.152.42@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fc224ec00, cur 1439193289 expire 1439193139 last 1439193062
Aug 10 08:54:49 cs04r-sc-mds03-01 kernel: Lustre: Skipped 1 previous similar message
Aug 10 09:02:58 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000: haven't heard from client 9f6daedd-2cb8-9884-0f05-94539c3fe7cc (at 10.144.152.42@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881023a6b400, cur 1439193778 expire 1439193628 last 1439193551
Aug 10 09:02:58 cs04r-sc-mds03-01 kernel: Lustre: Skipped 1 previous similar message
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: ptlrpcd_22: page allocation failure. order:2, mode:0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Pid: 2686, comm: ptlrpcd_22 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa03102e2>] ? bnx2x_napi_disable_cnic+0xf2/0x120 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113ed23>] ? shrink_inactive_list+0x313/0x830
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8106d504>] ? enqueue_task_fair+0x64/0x100
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8105c1e6>] ? enqueue_task+0x66/0x80
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810586cd>] ? check_preempt_curr+0x6d/0x90
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810649fe>] ? try_to_wake_up+0x24e/0x3e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: ptlrpcd_6: page allocation failure. order:2, mode:0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Pid: 2670, comm: ptlrpcd_6 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: ptlrpcd_6: page allocation failure. order:2, mode:0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Pid: 2670, comm: ptlrpcd_6 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: ptlrpcd_14: page allocation failure. order:2, mode:0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Pid: 2678, comm: ptlrpcd_14 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: ptlrpcd_6: page allocation failure. order:2, mode:0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Pid: 2670, comm: ptlrpcd_6 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810644f8>] ? load_balance_fair+0x208/0x300
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: ptlrpcd_14: page allocation failure. order:2, mode:0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Pid: 2678, comm: ptlrpcd_14 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810644f8>] ? load_balance_fair+0x208/0x300
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: ptlrpcd_6: page allocation failure. order:2, mode:0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Pid: 2670, comm: ptlrpcd_6 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810577e9>] ? __wake_up_common+0x59/0x90
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810107f6>] ? alloc_ldt+0x66/0x200
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810347c6>] ? lapic_suspend+0x86/0x1d0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81033bcd>] ? lapic_next_event+0x1d/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810a0042>] ? bump_cpu_timer+0xb2/0xf0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 10:22:03 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: __ratelimit: 20 callbacks suppressed
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: ptlrpcd_8: page allocation failure. order:2, mode:0x20
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: Pid: 2672, comm: ptlrpcd_8 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: ptlrpcd_14: page allocation failure. order:2, mode:0x20
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: Pid: 2678, comm: ptlrpcd_14 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff814b28c4>] ? tcp_transmit_skb+0x4b4/0x8b0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: kiblnd_sd_01_03: page allocation failure. order:2, mode:0x20
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: Pid: 2629, comm: kiblnd_sd_01_03 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff810f018e>] ? rcu_start_gp+0x1be/0x230
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffffa1099868>] ? kiblnd_check_sends+0x288/0xea0 [ko2iblnd]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff810577e9>] ? __wake_up_common+0x59/0x90
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa109c9b1>] ? kiblnd_handle_rx+0x2a1/0x640 [ko2iblnd]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa109db7e>] ? kiblnd_scheduler+0xe2e/0x1000 [ko2iblnd]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffffa109cd50>] ? kiblnd_scheduler+0x0/0x1000 [ko2iblnd]
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 10:22:07 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: ptlrpcd_4: page allocation failure. order:2, mode:0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: ptlrpcd_28: page allocation failure. order:2, mode:0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: Pid: 2692, comm: ptlrpcd_28 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: ptlrpcd_12: page allocation failure. order:2, mode:0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: Pid: 2676, comm: ptlrpcd_12 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: ptlrpcd_28: page allocation failure. order:2, mode:0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: Pid: 2692, comm: ptlrpcd_28 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: Pid: 2668, comm: ptlrpcd_4 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: __ratelimit: 12 callbacks suppressed
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 10:22:11 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:12 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: ptlrpcd_18: page allocation failure. order:2, mode:0x20
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: Pid: 2682, comm: ptlrpcd_18 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100fbd9>] ? handle_irq+0x49/0xa0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d728>] ? irq_exit+0x48/0x90
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81033bcd>] ? lapic_next_event+0x1d/0x30
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff810b034a>] ? tick_program_event+0x2a/0x30
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff815344ca>] ? smp_apic_timer_interrupt+0x4a/0x60
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100bb93>] ? apic_timer_interrupt+0x13/0x20
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113e81e>] ? shrink_active_list+0x17e/0x370
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113f605>] ? shrink_mem_cgroup_zone+0x3c5/0x540
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 10:22:22 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff814b28c4>] ? tcp_transmit_skb+0x4b4/0x8b0
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff81175f52>] ? kfree+0x122/0x320
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffffa070d597>] ? lnet_me_unlink+0xc7/0x140 [lnet]
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffffa07111a0>] ? lnet_md_unlink+0x250/0x340 [lnet]
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0711356>] ? LNetMDUnlink+0xc6/0x160 [lnet]
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8d5a0>] ? ptlrpc_unregister_reply+0x180/0x7f0 [ptlrpc]
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8f4d0>] ? ptlrpc_check_set+0x1300/0x1c60 [ptlrpc]
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:33 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: ptlrpcd_2: page allocation failure. order:2, mode:0x20
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: Pid: 2666, comm: ptlrpcd_2 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:36 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: ptlrpcd_2: page allocation failure. order:2, mode:0x20
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: Pid: 2666, comm: ptlrpcd_2 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:22:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:22:42 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c3500439a: rc = -2
Aug 10 10:22:42 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 2 previous similar messages
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffffa03153c7>] ? bnx2x_rx_int+0xfc7/0x1670 [bnx2x]
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 10:22:47 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff810108ca>] ? alloc_ldt+0x13a/0x200
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff8103d626>] ? fill_pte+0x36/0xf0
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff81033bcd>] ? lapic_next_event+0x1d/0x30
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff810a0042>] ? bump_cpu_timer+0xb2/0xf0
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 10:22:49 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff8106939c>] ? rebalance_domains+0x56c/0x6a0
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 10:22:59 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: ptlrpcd_12: page allocation failure. order:2, mode:0x20
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: Pid: 2676, comm: ptlrpcd_12 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff810b0275>] ? tick_dev_program_event+0x65/0xc0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: kswapd0: page allocation failure. order:2, mode:0x20
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: Pid: 340, comm: kswapd0 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: ptlrpcd_6: page allocation failure. order:2, mode:0x20
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: Pid: 2670, comm: ptlrpcd_6 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113ed23>] ? shrink_inactive_list+0x313/0x830
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff811ab48b>] ? iput+0x3b/0x70
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa1772df4>] ? osd_object_delete+0x1d4/0x2f0 [osd_ldiskfs]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa082fcb1>] ? lu_object_free+0x81/0x1a0 [obdclass]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7452>] ? cfs_hash_bd_from_key+0x42/0xd0 [libcfs]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0830c07>] ? lu_site_purge+0x2e7/0x4f0 [obdclass]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0830f98>] ? lu_cache_shrink+0x188/0x310 [obdclass]
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113d84a>] ? shrink_slab+0x8a/0x1a0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81140c5a>] ? balance_pgdat+0x57a/0x800
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81141014>] ? kswapd+0x134/0x3b0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff81140ee0>] ? kswapd+0x0/0x3b0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:23:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: ptlrpcd_4: page allocation failure. order:2, mode:0x20
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: Pid: 2668, comm: ptlrpcd_4 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff810b0275>] ? tick_dev_program_event+0x65/0xc0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 10:23:04 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: lfsck: page allocation failure. order:2, mode:0x20
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: Pid: 38531, comm: lfsck Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff810577e9>] ? __wake_up_common+0x59/0x90
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffffa0e9aec5>] ? lustre_msg_add_version+0x75/0xc0 [ptlrpc]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0e9b440>] ? lustre_pack_request+0xb0/0x170 [ptlrpc]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0e86b95>] ? __ptlrpc_request_bufs_pack+0xe5/0x300 [ptlrpc]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0e86e0c>] ? ptlrpc_request_bufs_pack+0x5c/0x80 [ptlrpc]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0e86e54>] ? ptlrpc_request_pack+0x24/0x70 [ptlrpc]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0f0843b>] ? out_prep_update_req+0xdb/0x260 [ptlrpc]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa1b9d50f>] ? osp_unplug_async_request+0x3f/0x160 [osp]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa1b85349>] ? osp_declare_xattr_get+0x219/0x2a0 [osp]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa1b883b0>] ? osp_xattr_get_interpterer+0x0/0x510 [osp]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa1942b8e>] ? lfsck_layout_scan_stripes+0x126e/0x1870 [lfsck]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa1647cba>] ? ldiskfs_xattr_get+0x10a/0x330 [ldiskfs]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa1ab4ff4>] ? lod_attr_get+0xa4/0x140 [lod]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa1944015>] ? lfsck_layout_master_exec_oit+0x565/0xd00 [lfsck]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa083143f>] ? lu_object_find_try+0x1df/0x260 [obdclass]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa19144b3>] ? lfsck_exec_oit+0x73/0xb40 [lfsck]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa11271e3>] ? fld_server_lookup+0x53/0x330 [fld]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa19155fd>] ? lfsck_master_oit_engine+0x67d/0x1ef0 [lfsck]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa191794e>] ? lfsck_master_engine+0xade/0x13e0 [lfsck]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa1916e70>] ? lfsck_master_engine+0x0/0x13e0 [lfsck]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: lfsck: page allocation failure. order:2, mode:0x20
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: Pid: 38531, comm: lfsck Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff810b0760>] ? tick_sched_timer+0x0/0xc0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81039787>] ? native_apic_msr_write+0x37/0x40
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff810aed34>] ? clockevents_program_event+0x54/0xa0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff810b0275>] ? tick_dev_program_event+0x65/0xc0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffffa1771723>] ? osd_object_init+0x333/0x1830 [osd_ldiskfs]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa1aa113c>] ? lod_fld_lookup+0x16c/0x480 [lod]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa082e03c>] ? lu_object_add+0x2c/0x30 [obdclass]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa082fea8>] ? lu_object_alloc+0xd8/0x320 [obdclass]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa08313b1>] ? lu_object_find_try+0x151/0x260 [obdclass]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0831571>] ? lu_object_find_at+0xb1/0xe0 [obdclass]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa11271e3>] ? fld_server_lookup+0x53/0x330 [fld]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa08315df>] ? lu_object_find_slice+0x1f/0x80 [obdclass]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa19155ba>] ? lfsck_master_oit_engine+0x63a/0x1ef0 [lfsck]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa191794e>] ? lfsck_master_engine+0xade/0x13e0 [lfsck]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffffa1916e70>] ? lfsck_master_engine+0x0/0x13e0 [lfsck]
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:23:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: lfsck: page allocation failure. order:2, mode:0x20
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: Pid: 38531, comm: lfsck Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8152d777>] ? _spin_unlock_irqrestore+0x17/0x20
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff8105bd23>] ? __wake_up+0x53/0x70
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0e860a3>] ? ptlrpc_set_add_new_req+0xe3/0x160 [ptlrpc]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbbab>] ? ptlrpcd_add_req+0x1fb/0x2b0 [ptlrpc]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0e99657>] ? ptlrpc_request_set_replen+0x37/0x60 [ptlrpc]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0f084c8>] ? out_prep_update_req+0x168/0x260 [ptlrpc]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa1b9d5fa>] ? osp_unplug_async_request+0x12a/0x160 [osp]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa1b85349>] ? osp_declare_xattr_get+0x219/0x2a0 [osp]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa1b883b0>] ? osp_xattr_get_interpterer+0x0/0x510 [osp]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa1942b8e>] ? lfsck_layout_scan_stripes+0x126e/0x1870 [lfsck]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa1647cba>] ? ldiskfs_xattr_get+0x10a/0x330 [ldiskfs]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa1ab4ff4>] ? lod_attr_get+0xa4/0x140 [lod]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa1944015>] ? lfsck_layout_master_exec_oit+0x565/0xd00 [lfsck]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa083143f>] ? lu_object_find_try+0x1df/0x260 [obdclass]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa19144b3>] ? lfsck_exec_oit+0x73/0xb40 [lfsck]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa11271e3>] ? fld_server_lookup+0x53/0x330 [fld]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa19155fd>] ? lfsck_master_oit_engine+0x67d/0x1ef0 [lfsck]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa191794e>] ? lfsck_master_engine+0xade/0x13e0 [lfsck]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffffa1916e70>] ? lfsck_master_engine+0x0/0x13e0 [lfsck]
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:23:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:23:41 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c35005159: rc = -2
Aug 10 10:23:41 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 31 previous similar messages
Aug 10 10:23:56 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c350010a5: rc = -2
Aug 10 10:23:56 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 15 previous similar messages
Aug 10 10:23:59 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c35001f7f: rc = -2
Aug 10 10:23:59 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 72 previous similar messages
Aug 10 10:24:10 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c35000ea2: rc = -2
Aug 10 10:24:10 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 102 previous similar messages
Aug 10 10:24:19 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c35001e20: rc = -2
Aug 10 10:24:19 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 187 previous similar messages
Aug 10 10:24:36 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c35001090: rc = -2
Aug 10 10:24:36 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 367 previous similar messages
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8145092b>] ? consume_skb+0x3b/0x80
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8145dbdd>] ? dev_kfree_skb_any+0x3d/0x50
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa03122a1>] ? bnx2x_msix_fp_int+0xd1/0x170 [bnx2x]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:29:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff812a2bb8>] ? swiotlb_dma_mapping_error+0x18/0x30
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0310302>] ? bnx2x_napi_disable_cnic+0x112/0x120 [bnx2x]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff811757cf>] ? kmem_cache_free+0xbf/0x2b0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811c3a22>] ? free_buffer_head+0x22/0x50
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811c3df9>] ? try_to_free_buffers+0x79/0xc0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0233d57>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8117ec51>] ? __mem_cgroup_uncharge_common+0x81/0x300
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa163c6f8>] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113d228>] ? __remove_mapping+0xb8/0x160
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811cb726>] ? blkdev_releasepage+0x36/0x50
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811239f0>] ? try_to_release_page+0x30/0x60
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113e29f>] ? shrink_page_list.clone.3+0x5af/0x680
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa03102e2>] ? bnx2x_napi_disable_cnic+0xf2/0x120 [bnx2x]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113ed23>] ? shrink_inactive_list+0x313/0x830
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:29:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: mdt01_002: page allocation failure. order:2, mode:0x20
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: Pid: 2853, comm: mdt01_002 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5bd5>] ? null_alloc_rs+0xc5/0x390 [ptlrpc]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5bd5>] ? null_alloc_rs+0xc5/0x390 [ptlrpc]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4684>] ? sptlrpc_svc_alloc_rs+0x74/0x360 [ptlrpc]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0e9afad>] ? lustre_pack_reply_v2+0x9d/0x280 [ptlrpc]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0e9b236>] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0e9b381>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec1ec3>] ? req_capsule_server_pack+0x53/0x100 [ptlrpc]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa19f1bf5>] ? mdt_getxattr+0x635/0x1470 [mdt]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa19d02c5>] ? mdt_object_lock_internal+0x65/0x360 [mdt]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa19d072c>] ? mdt_intent_getxattr+0x9c/0x150 [mdt]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa08315b6>] ? lu_object_find+0x16/0x20 [obdclass]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa19c7cf4>] ? mdt_intent_policy+0x494/0xce0 [mdt]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0e4f4f9>] ? ldlm_lock_enqueue+0x129/0x9d0 [ptlrpc]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0e7b46b>] ? ldlm_handle_enqueue0+0x51b/0x13f0 [ptlrpc]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa05e3c8a>] ? lc_watchdog_touch+0x7a/0x190 [libcfs]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0efb921>] ? tgt_enqueue+0x61/0x230 [ptlrpc]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0efc56e>] ? tgt_request_handle+0x8be/0x1000 [ptlrpc]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0eac5a1>] ? ptlrpc_main+0xe41/0x1960 [ptlrpc]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8106c4f0>] ? pick_next_task_fair+0xd0/0x130
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0eab760>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:29:10 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: ptlrpcd_4: page allocation failure. order:2, mode:0x20
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: Pid: 2668, comm: ptlrpcd_4 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: mdt01_023: page allocation failure. order:2, mode:0x20
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: Pid: 3671, comm: mdt01_023 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffffa0efbde8>] ? tgt_request_handle+0x138/0x1000 [ptlrpc]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0eac5a1>] ? ptlrpc_main+0xe41/0x1960 [ptlrpc]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8106c4f0>] ? pick_next_task_fair+0xd0/0x130
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0eab760>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:29:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: ptlrpcd_22: page allocation failure. order:2, mode:0x20
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: Pid: 2686, comm: ptlrpcd_22 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:29:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: ptlrpcd_4: page allocation failure. order:2, mode:0x20
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: Pid: 2668, comm: ptlrpcd_4 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff81039787>] ? native_apic_msr_write+0x37/0x40
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:29:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:29:26 cs04r-sc-mds03-01 kernel: ptlrpcd_18: page allocation failure. order:2, mode:0x20
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: Pid: 2682, comm: ptlrpcd_18 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa03122a1>] ? bnx2x_msix_fp_int+0xd1/0x170 [bnx2x]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff810b034a>] ? tick_program_event+0x2a/0x30
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff815344ca>] ? smp_apic_timer_interrupt+0x4a/0x60
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8100bb93>] ? apic_timer_interrupt+0x13/0x20
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113ed23>] ? shrink_inactive_list+0x313/0x830
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:29:27 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 10:29:36 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: ptlrpcd_12: page allocation failure. order:2, mode:0x20
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: Pid: 2676, comm: ptlrpcd_12 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff810649fe>] ? try_to_wake_up+0x24e/0x3e0
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:29:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:32:37 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c35009a44: rc = -2
Aug 10 10:32:37 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 578 previous similar messages
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:35:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: ptlrpcd_6: page allocation failure. order:2, mode:0x20
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: Pid: 2670, comm: ptlrpcd_6 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113ed23>] ? shrink_inactive_list+0x313/0x830
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81461030>] ? napi_skb_finish+0x50/0x70
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81462909>] ? napi_gro_receive+0x39/0x50
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa03153c7>] ? bnx2x_rx_int+0xfc7/0x1670 [bnx2x]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff811757cf>] ? kmem_cache_free+0xbf/0x2b0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811c3a22>] ? free_buffer_head+0x22/0x50
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811c3df9>] ? try_to_free_buffers+0x79/0xc0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0233d57>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8117ec51>] ? __mem_cgroup_uncharge_common+0x81/0x300
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa163c6f8>] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8113d228>] ? __remove_mapping+0xb8/0x160
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811cb726>] ? blkdev_releasepage+0x36/0x50
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811239f0>] ? try_to_release_page+0x30/0x60
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8113e29f>] ? shrink_page_list.clone.3+0x5af/0x680
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:35:09 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: OI_scrub: page allocation failure. order:2, mode:0x20
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: Pid: 38532, comm: OI_scrub Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113b945>] ? mark_page_accessed+0x5/0x50
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff811c4a97>] ? __find_get_block+0x97/0xe0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff811c4b04>] ? __getblk+0x24/0x2a0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff811c4b0c>] ? __getblk+0x2c/0x2a0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa161597e>] ? ldiskfs_get_branch+0xee/0x130 [ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1618874>] ? ldiskfs_ind_get_blocks+0xc4/0x610 [ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1618b0e>] ? ldiskfs_ind_get_blocks+0x35e/0x610 [ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1619158>] ? ldiskfs_get_blocks+0x168/0x2b0 [ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff811c4a97>] ? __find_get_block+0x97/0xe0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1619879>] ? ldiskfs_getblk+0x79/0x1f0 [ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff811c4b0c>] ? __getblk+0x2c/0x2a0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1619a08>] ? ldiskfs_bread+0x18/0x80 [ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1646dde>] ? ldiskfs_xattr_find_entry+0x3e/0x120 [ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa178061a>] ? iam_node_read+0x5a/0xa0 [osd_ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1780d43>] ? __iam_path_lookup+0x63/0x1f0 [osd_ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa17811f3>] ? __iam_it_get+0x93/0x1d0 [osd_ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa178151e>] ? iam_it_get+0x2e/0x150 [osd_ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa177cc68>] ? __osd_oi_lookup+0x118/0x3b0 [osd_ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa177cf84>] ? osd_oi_lookup+0x84/0x150 [osd_ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1797331>] ? osd_scrub_exec+0xa11/0x1ab0 [osd_ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1794152>] ? osd_scrub_next+0x222/0x470 [osd_ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1798899>] ? osd_inode_iteration+0x4c9/0xd50 [osd_ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1796920>] ? osd_scrub_exec+0x0/0x1ab0 [osd_ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1793f30>] ? osd_scrub_next+0x0/0x470 [osd_ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa179a245>] ? osd_scrub_main+0x875/0xed0 [osd_ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa17999d0>] ? osd_scrub_main+0x0/0xed0 [osd_ldiskfs]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: ptlrpcd_14: page allocation failure. order:2, mode:0x20
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: Pid: 2678, comm: ptlrpcd_14 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:35:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:36:32 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c350087dd: rc = -2
Aug 10 10:40:47 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200002b1400728c: rc = -2
Aug 10 10:40:47 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 1 previous similar message
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: ptlrpcd_0: page allocation failure. order:2, mode:0x20
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: Pid: 2664, comm: ptlrpcd_0 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:42:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: ptlrpcd_8: page allocation failure. order:2, mode:0x20
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: Pid: 2672, comm: ptlrpcd_8 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8105022a>] ? memtype_seq_next+0xa/0x20
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:42:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: ptlrpcd_18: page allocation failure. order:2, mode:0x20
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: Pid: 2682, comm: ptlrpcd_18 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff814b28c4>] ? tcp_transmit_skb+0x4b4/0x8b0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:42:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:52:18 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c35008762: rc = -2
Aug 10 10:52:18 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 252 previous similar messages
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: ptlrpcd_18: page allocation failure. order:2, mode:0x20
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: Pid: 2682, comm: ptlrpcd_18 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: ptlrpcd_26: page allocation failure. order:2, mode:0x20
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: Pid: 2690, comm: ptlrpcd_26 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8149db38>] ? ip_output+0xb8/0xc0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8149ce15>] ? ip_local_out+0x25/0x30
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff810577e9>] ? __wake_up_common+0x59/0x90
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81064557>] ? load_balance_fair+0x267/0x300
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 10:53:35 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:03:03 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c35000a85: rc = -2
Aug 10 11:03:03 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 102 previous similar messages
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:08:03 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: ptlrpcd_8: page allocation failure. order:2, mode:0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: Pid: 2672, comm: ptlrpcd_8 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:08:04 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff812a2bb8>] ? swiotlb_dma_mapping_error+0x18/0x30
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa03102e2>] ? bnx2x_napi_disable_cnic+0xf2/0x120 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: ptlrpcd_8: page allocation failure. order:2, mode:0x20
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: Pid: 2672, comm: ptlrpcd_8 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa03101ea>] ? bnx2x_drain_tx_queues+0x13a/0x140 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa082d225>] ? lu_context_exit+0x65/0xa0 [obdclass]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:08:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: __ratelimit: 14 callbacks suppressed
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: ptlrpcd_26: page allocation failure. order:2, mode:0x20
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: Pid: 2690, comm: ptlrpcd_26 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa03153c7>] ? bnx2x_rx_int+0xfc7/0x1670 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81461030>] ? napi_skb_finish+0x50/0x70
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa03122a1>] ? bnx2x_msix_fp_int+0xd1/0x170 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: ptlrpcd_26: page allocation failure. order:2, mode:0x20
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: Pid: 2690, comm: ptlrpcd_26 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa03153c7>] ? bnx2x_rx_int+0xfc7/0x1670 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81461030>] ? napi_skb_finish+0x50/0x70
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa03122a1>] ? bnx2x_msix_fp_int+0xd1/0x170 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:09:16 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa03102e2>] ? bnx2x_napi_disable_cnic+0xf2/0x120 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:09:17 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: ptlrpcd_8: page allocation failure. order:2, mode:0x20
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: Pid: 2672, comm: ptlrpcd_8 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: ptlrpcd_26: page allocation failure. order:2, mode:0x20
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8149778d>] ? ip_local_deliver_finish+0xdd/0x2d0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa031460f>] ? bnx2x_rx_int+0x20f/0x1670 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff812a3530>] ? swiotlb_map_page+0x0/0x100
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: Pid: 2690, comm: ptlrpcd_26 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff814b28c4>] ? tcp_transmit_skb+0x4b4/0x8b0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: __ratelimit: 12 callbacks suppressed
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: ptlrpcd_26: page allocation failure. order:2, mode:0x20
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: Pid: 2690, comm: ptlrpcd_26 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa03153c7>] ? bnx2x_rx_int+0xfc7/0x1670 [bnx2x]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa03122a1>] ? bnx2x_msix_fp_int+0xd1/0x170 [bnx2x]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: ptlrpcd_12: page allocation failure. order:2, mode:0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Pid: 2676, comm: ptlrpcd_12 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8149778d>] ? ip_local_deliver_finish+0xdd/0x2d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa031460f>] ? bnx2x_rx_int+0x20f/0x1670 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff812a3530>] ? swiotlb_map_page+0x0/0x100
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81138937>] ? get_dirty_limits+0x27/0x2f0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: ptlrpcd_24: page allocation failure. order:2, mode:0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Pid: 2688, comm: ptlrpcd_24 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: ptlrpcd_10: page allocation failure. order:2, mode:0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Pid: 2674, comm: ptlrpcd_10 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8149ce15>] ? ip_local_out+0x25/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8149d310>] ? ip_queue_xmit+0x190/0x420
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: ptlrpcd_24: page allocation failure. order:2, mode:0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Pid: 2688, comm: ptlrpcd_24 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa031460f>] ? bnx2x_rx_int+0x20f/0x1670 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff812a3530>] ? swiotlb_map_page+0x0/0x100
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810577e9>] ? __wake_up_common+0x59/0x90
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:09:25 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: __ratelimit: 13 callbacks suppressed
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: ptlrpcd_2: page allocation failure. order:2, mode:0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: Pid: 2666, comm: ptlrpcd_2 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: ptlrpcd_14: page allocation failure. order:2, mode:0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: Pid: 2678, comm: ptlrpcd_14 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff814b28c4>] ? tcp_transmit_skb+0x4b4/0x8b0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: ptlrpcd_4: page allocation failure. order:2, mode:0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: Pid: 2668, comm: ptlrpcd_4 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: ptlrpcd_22: page allocation failure. order:2, mode:0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: Pid: 2686, comm: ptlrpcd_22 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8149db38>] ? ip_output+0xb8/0xc0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8149ce15>] ? ip_local_out+0x25/0x30
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8149d310>] ? ip_queue_xmit+0x190/0x420
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:11:42 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:12:01 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa03153c7>] ? bnx2x_rx_int+0xfc7/0x1670 [bnx2x]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa03122a1>] ? bnx2x_msix_fp_int+0xd1/0x170 [bnx2x]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: ptlrpcd_18: page allocation failure. order:2, mode:0x20
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: Pid: 2682, comm: ptlrpcd_18 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0310182>] ? bnx2x_drain_tx_queues+0xd2/0x140 [bnx2x]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:12:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:14:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: lfsck: page allocation failure. order:2, mode:0x20
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: Pid: 38531, comm: lfsck Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff81134da0>] ? __free_pages+0x60/0xa0
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff8117355b>] ? kmem_freepages+0xeb/0x130
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff81175ba4>] ? free_block+0x154/0x170
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8152d722>] ? _spin_lock+0x12/0x30
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec2a08>] ? sptlrpc_import_sec_ref+0x28/0x50 [ptlrpc]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec958c>] ? import_sec_validate_get+0x3c/0x260 [ptlrpc]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec97ff>] ? sptlrpc_req_get_ctx+0x4f/0x260 [ptlrpc]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0e86b12>] ? __ptlrpc_request_bufs_pack+0x62/0x300 [ptlrpc]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa1ab42cc>] ? lod_object_init+0x22c/0x3c0 [lod]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0e86e0c>] ? ptlrpc_request_bufs_pack+0x5c/0x80 [ptlrpc]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0e86e54>] ? ptlrpc_request_pack+0x24/0x70 [ptlrpc]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa0f0843b>] ? out_prep_update_req+0xdb/0x260 [ptlrpc]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa1b9d50f>] ? osp_unplug_async_request+0x3f/0x160 [osp]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa1b85349>] ? osp_declare_xattr_get+0x219/0x2a0 [osp]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa1b883b0>] ? osp_xattr_get_interpterer+0x0/0x510 [osp]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa1942b8e>] ? lfsck_layout_scan_stripes+0x126e/0x1870 [lfsck]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa1647cba>] ? ldiskfs_xattr_get+0x10a/0x330 [ldiskfs]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa1ab4ff4>] ? lod_attr_get+0xa4/0x140 [lod]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa1944015>] ? lfsck_layout_master_exec_oit+0x565/0xd00 [lfsck]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa083143f>] ? lu_object_find_try+0x1df/0x260 [obdclass]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa19144b3>] ? lfsck_exec_oit+0x73/0xb40 [lfsck]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa11271e3>] ? fld_server_lookup+0x53/0x330 [fld]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa19155fd>] ? lfsck_master_oit_engine+0x67d/0x1ef0 [lfsck]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa191794e>] ? lfsck_master_engine+0xade/0x13e0 [lfsck]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffffa1916e70>] ? lfsck_master_engine+0x0/0x13e0 [lfsck]
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:14:10 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:18:58 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c350056e1: rc = -2
Aug 10 11:18:58 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 44 previous similar messages
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: ptlrpcd_8: page allocation failure. order:2, mode:0x20
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: Pid: 2672, comm: ptlrpcd_8 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa03153c7>] ? bnx2x_rx_int+0xfc7/0x1670 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa03122a1>] ? bnx2x_msix_fp_int+0xd1/0x170 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: ptlrpcd_0: page allocation failure. order:2, mode:0x20
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: Pid: 2664, comm: ptlrpcd_0 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b7438>] ? smp_call_function_many+0x1e8/0x260
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:21:18 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: ptlrpcd_12: page allocation failure. order:2, mode:0x20
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: Pid: 2676, comm: ptlrpcd_12 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: ptlrpcd_14: page allocation failure. order:2, mode:0x20
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: Pid: 2678, comm: ptlrpcd_14 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:21:19 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: ptlrpcd_4: page allocation failure. order:2, mode:0x20
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: Pid: 2668, comm: ptlrpcd_4 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8152d722>] ? _spin_lock+0x12/0x30
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811c3dc5>] ? try_to_free_buffers+0x45/0xc0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0233d57>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa163c6f8>] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8113d228>] ? __remove_mapping+0xb8/0x160
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811cb726>] ? blkdev_releasepage+0x36/0x50
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811239f0>] ? try_to_release_page+0x30/0x60
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8113e29f>] ? shrink_page_list.clone.3+0x5af/0x680
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: ptlrpcd_24: page allocation failure. order:2, mode:0x20
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: Pid: 2688, comm: ptlrpcd_24 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa03102fa>] ? bnx2x_napi_disable_cnic+0x10a/0x120 [bnx2x]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:21:20 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa03153c7>] ? bnx2x_rx_int+0xfc7/0x1670 [bnx2x]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa03122a1>] ? bnx2x_msix_fp_int+0xd1/0x170 [bnx2x]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8106d504>] ? enqueue_task_fair+0x64/0x100
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8105c1e6>] ? enqueue_task+0x66/0x80
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff810586cd>] ? check_preempt_curr+0x6d/0x90
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff810649fe>] ? try_to_wake_up+0x24e/0x3e0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: ptlrpcd_14: page allocation failure. order:2, mode:0x20
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: Pid: 2678, comm: ptlrpcd_14 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff814b28c4>] ? tcp_transmit_skb+0x4b4/0x8b0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:22:43 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: ptlrpcd_26: page allocation failure. order:2, mode:0x20
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: Pid: 2690, comm: ptlrpcd_26 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81134da0>] ? __free_pages+0x60/0xa0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8117355b>] ? kmem_freepages+0xeb/0x130
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff811759f3>] ? slab_destroy+0x33/0x90
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81175ab0>] ? free_block+0x60/0x170
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113ef99>] ? shrink_inactive_list+0x589/0x830
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:22:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: ptlrpcd_28: page allocation failure. order:2, mode:0x20
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: Pid: 2692, comm: ptlrpcd_28 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa03153c7>] ? bnx2x_rx_int+0xfc7/0x1670 [bnx2x]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:22:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: ptlrpcd_0: page allocation failure. order:2, mode:0x20
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: Pid: 2664, comm: ptlrpcd_0 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: lfsck: page allocation failure. order:2, mode:0x20
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: Pid: 38531, comm: lfsck Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8152acb2>] ? _cond_resched+0x12/0x40
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff811c4b0c>] ? __getblk+0x2c/0x2a0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa161597e>] ? ldiskfs_get_branch+0xee/0x130 [ldiskfs]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa1618874>] ? ldiskfs_ind_get_blocks+0xc4/0x610 [ldiskfs]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa1619158>] ? ldiskfs_get_blocks+0x168/0x2b0 [ldiskfs]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff811c4b0c>] ? __getblk+0x2c/0x2a0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa1619879>] ? ldiskfs_getblk+0x79/0x1f0 [ldiskfs]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e86d36>] ? __ptlrpc_request_bufs_pack+0x286/0x300 [ptlrpc]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e9a235>] ? lustre_msg_buf+0x55/0x60 [ptlrpc]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa1619a08>] ? ldiskfs_bread+0x18/0x80 [ldiskfs]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa178061a>] ? iam_node_read+0x5a/0xa0 [osd_ldiskfs]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa1780d43>] ? __iam_path_lookup+0x63/0x1f0 [osd_ldiskfs]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa17811f3>] ? __iam_it_get+0x93/0x1d0 [osd_ldiskfs]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173efb>] ? cache_alloc_refill+0x15b/0x240
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa178151e>] ? iam_it_get+0x2e/0x150 [osd_ldiskfs]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa177cc68>] ? __osd_oi_lookup+0x118/0x3b0 [osd_ldiskfs]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa177cf84>] ? osd_oi_lookup+0x84/0x150 [osd_ldiskfs]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa17718d9>] ? osd_object_init+0x4e9/0x1830 [osd_ldiskfs]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa1aa113c>] ? lod_fld_lookup+0x16c/0x480 [lod]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa082e03c>] ? lu_object_add+0x2c/0x30 [obdclass]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa082fea8>] ? lu_object_alloc+0xd8/0x320 [obdclass]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa08313b1>] ? lu_object_find_try+0x151/0x260 [obdclass]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0831571>] ? lu_object_find_at+0xb1/0xe0 [obdclass]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa11271e3>] ? fld_server_lookup+0x53/0x330 [fld]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa08315df>] ? lu_object_find_slice+0x1f/0x80 [obdclass]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa19155ba>] ? lfsck_master_oit_engine+0x63a/0x1ef0 [lfsck]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa191794e>] ? lfsck_master_engine+0xade/0x13e0 [lfsck]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa1916e70>] ? lfsck_master_engine+0x0/0x13e0 [lfsck]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113e489>] ? move_active_pages_to_lru+0x119/0x1c0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113e922>] ? shrink_active_list+0x282/0x370
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113f605>] ? shrink_mem_cgroup_zone+0x3c5/0x540
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:24:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: ptlrpcd_28: page allocation failure. order:2, mode:0x20
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: Pid: 2692, comm: ptlrpcd_28 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8106d175>] ? enqueue_entity+0x125/0x450
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8106d504>] ? enqueue_task_fair+0x64/0x100
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:24:07 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: ptlrpcd_2: page allocation failure. order:2, mode:0x20
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: Pid: 2666, comm: ptlrpcd_2 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:24:47 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: ptlrpcd_14: page allocation failure. order:2, mode:0x20
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: Pid: 2678, comm: ptlrpcd_14 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:28:05 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: lfsck_layout: page allocation failure. order:2, mode:0x20
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: Pid: 38533, comm: lfsck_layout Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e9b440>] ? lustre_pack_request+0xb0/0x170 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e9040e>] ? ptlrpc_set_wait+0x5de/0x890 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e9d750>] ? lustre_swab_object_update_request+0x0/0xb0 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e9adc6>] ? lustre_msg_set_jobid+0xb6/0x140 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e90741>] ? ptlrpc_queue_wait+0x81/0x220 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0f086d1>] ? out_remote_sync+0x111/0x200 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa1b84a92>] ? osp_attr_get+0x352/0x600 [osp]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa1951e50>] ? lfsck_layout_assistant_handler_p1+0x530/0x19f0 [lfsck]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa19186e6>] ? lfsck_assistant_engine+0x496/0x1de0 [lfsck]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa1918250>] ? lfsck_assistant_engine+0x0/0x1de0 [lfsck]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8117eee1>] ? mem_cgroup_uncharge_cache_page+0x11/0x20
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113d228>] ? __remove_mapping+0xb8/0x160
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:28:06 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: ptlrpcd_6: page allocation failure. order:2, mode:0x20
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: Pid: 2670, comm: ptlrpcd_6 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064557>] ? load_balance_fair+0x267/0x300
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: ptlrpcd_6: page allocation failure. order:2, mode:0x20
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: ptlrpcd_2: page allocation failure. order:2, mode:0x20
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: Pid: 2666, comm: ptlrpcd_2 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0310272>] ? bnx2x_napi_disable_cnic+0x82/0x120 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: Pid: 2670, comm: ptlrpcd_6 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064557>] ? load_balance_fair+0x267/0x300
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:28:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:31:34 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c3500aee3: rc = -2
Aug 10 11:31:34 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 5197 previous similar messages
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: ptlrpcd_24: page allocation failure. order:2, mode:0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: Pid: 2688, comm: ptlrpcd_24 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810577e9>] ? __wake_up_common+0x59/0x90
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: ptlrpcd_12: page allocation failure. order:2, mode:0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: Pid: 2676, comm: ptlrpcd_12 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810577e9>] ? __wake_up_common+0x59/0x90
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: ptlrpcd_12: page allocation failure. order:2, mode:0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: Pid: 2676, comm: ptlrpcd_12 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810577e9>] ? __wake_up_common+0x59/0x90
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: ptlrpcd_8: page allocation failure. order:2, mode:0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: Pid: 2672, comm: ptlrpcd_8 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: ptlrpcd_8: page allocation failure. order:2, mode:0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: Pid: 2672, comm: ptlrpcd_8 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffffa023c8de>] ? jbd2_journal_grab_journal_head+0xe/0x40 [jbd2]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0233cf8>] ? jbd2_journal_try_to_free_buffers+0x48/0x150 [jbd2]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8117ec51>] ? __mem_cgroup_uncharge_common+0x81/0x300
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa163c6f8>] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113d228>] ? __remove_mapping+0xb8/0x160
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811cb726>] ? blkdev_releasepage+0x36/0x50
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811239f0>] ? try_to_release_page+0x30/0x60
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113e29f>] ? shrink_page_list.clone.3+0x5af/0x680
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:45:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffffa03153c7>] ? bnx2x_rx_int+0xfc7/0x1670 [bnx2x]
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffffa03122a1>] ? bnx2x_msix_fp_int+0xd1/0x170 [bnx2x]
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:46:58 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffffa0312a8a>] ? bnx2x_start_xmit+0x31a/0x1690 [bnx2x]
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffffa0310182>] ? bnx2x_drain_tx_queues+0xd2/0x140 [bnx2x]
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:47:54 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: ptlrpcd_26: page allocation failure. order:2, mode:0x20
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: Pid: 2690, comm: ptlrpcd_26 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:49:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: ptlrpcd_2: page allocation failure. order:2, mode:0x20
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: Pid: 2666, comm: ptlrpcd_2 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:49:33 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: ptlrpcd_24: page allocation failure. order:2, mode:0x20
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: Pid: 2688, comm: ptlrpcd_24 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113e81e>] ? shrink_active_list+0x17e/0x370
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8113f605>] ? shrink_mem_cgroup_zone+0x3c5/0x540
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:52:51 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff8149db38>] ? ip_output+0xb8/0xc0
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff8149ce15>] ? ip_local_out+0x25/0x30
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff810577e9>] ? __wake_up_common+0x59/0x90
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:53:16 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: ptlrpcd_0: page allocation failure. order:2, mode:0x20
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: Pid: 2664, comm: ptlrpcd_0 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff810b0760>] ? tick_sched_timer+0x0/0xc0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81039787>] ? native_apic_msr_write+0x37/0x40
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff810aed34>] ? clockevents_program_event+0x54/0xa0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff810b0275>] ? tick_dev_program_event+0x65/0xc0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff810b034a>] ? tick_program_event+0x2a/0x30
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff810a375b>] ? hrtimer_interrupt+0x14b/0x260
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 11:53:31 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: kswapd0: page allocation failure. order:2, mode:0x20
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: Pid: 340, comm: kswapd0 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff810104ce>] ? sys_iopl+0x2e/0x80
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff81030dcb>] ? setup_p4_watchdog+0x15b/0x1f0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff81033bcd>] ? lapic_next_event+0x1d/0x30
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff810a0042>] ? bump_cpu_timer+0xb2/0xf0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8109ea55>] ? __wake_up_bit+0x15/0x40
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff8109eaef>] ? wake_up_bit+0x2f/0x40
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff811abf4f>] ? dispose_list+0xaf/0x120
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff811ac234>] ? shrink_icache_memory+0x274/0x2e0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff8113d8ea>] ? shrink_slab+0x12a/0x1a0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff81140c5a>] ? balance_pgdat+0x57a/0x800
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff81141014>] ? kswapd+0x134/0x3b0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff81140ee0>] ? kswapd+0x0/0x3b0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:54:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: ptlrpcd_8: page allocation failure. order:2, mode:0x20
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: Pid: 2672, comm: ptlrpcd_8 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:54:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: ptlrpcd_22: page allocation failure. order:2, mode:0x20
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: Pid: 2686, comm: ptlrpcd_22 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:55:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 11:59:37 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 11:59:40 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200000401000016: rc = -2
Aug 10 11:59:40 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 226 previous similar messages
Aug 10 12:02:22 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c3500825d: rc = -2
Aug 10 12:02:22 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 7 previous similar messages
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 12:03:01 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: ptlrpcd_0: page allocation failure. order:2, mode:0x20
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: Pid: 2664, comm: ptlrpcd_0 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff810b0760>] ? tick_sched_timer+0x0/0xc0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: ptlrpcd_24: page allocation failure. order:2, mode:0x20
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: Pid: 2688, comm: ptlrpcd_24 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: ptlrpcd_28: page allocation failure. order:2, mode:0x20
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: Pid: 2692, comm: ptlrpcd_28 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:03:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: ptlrpcd_0: page allocation failure. order:2, mode:0x20
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: Pid: 2664, comm: ptlrpcd_0 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: ptlrpcd_18: page allocation failure. order:2, mode:0x20
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: Pid: 2682, comm: ptlrpcd_18 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: ptlrpcd_10: page allocation failure. order:2, mode:0x20
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: Pid: 2674, comm: ptlrpcd_10 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:05:18 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: ptlrpcd_18: page allocation failure. order:2, mode:0x20
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: Pid: 2682, comm: ptlrpcd_18 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:10:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: ptlrpcd_28: page allocation failure. order:2, mode:0x20
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: Pid: 2692, comm: ptlrpcd_28 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff814b28c4>] ? tcp_transmit_skb+0x4b4/0x8b0
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:11:38 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff810501ea>] ? memtype_get_idx+0xca/0x100
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 12:12:43 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: ptlrpcd_2: page allocation failure. order:2, mode:0x20
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: Pid: 2666, comm: ptlrpcd_2 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8149ce15>] ? ip_local_out+0x25/0x30
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8149d310>] ? ip_queue_xmit+0x190/0x420
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:12:44 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff810577e9>] ? __wake_up_common+0x59/0x90
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81010868>] ? alloc_ldt+0xd8/0x200
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81034fb6>] ? default_send_IPI_mask_allbutself_phys+0xe6/0x120
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81033bcd>] ? lapic_next_event+0x1d/0x30
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff810a0042>] ? bump_cpu_timer+0xb2/0xf0
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 12:15:16 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: OI_scrub: page allocation failure. order:2, mode:0x20
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: Pid: 38532, comm: OI_scrub Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa03153c7>] ? bnx2x_rx_int+0xfc7/0x1670 [bnx2x]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa03122a1>] ? bnx2x_msix_fp_int+0xd1/0x170 [bnx2x]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812720dc>] ? blk_queue_bio+0x14c/0x610
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff81280af6>] ? throtl_find_tg+0x46/0x60
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff81281dea>] ? blk_throtl_bio+0x1ea/0x5f0
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa0003a5d>] ? dm_request+0x15d/0x1b0 [dm_mod]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff812710bf>] ? generic_make_request+0x29f/0x5f0
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff81126eb5>] ? mempool_alloc_slab+0x15/0x20
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff81127053>] ? mempool_alloc+0x63/0x140
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff811c4b7e>] ? __getblk+0x9e/0x2a0
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff81271480>] ? submit_bio+0x70/0x120
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff811c3f5d>] ? submit_bh+0x11d/0x1f0
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff811c5a33>] ? ll_rw_block+0x143/0x150
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa1619a33>] ? ldiskfs_bread+0x43/0x80 [ldiskfs]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa178061a>] ? iam_node_read+0x5a/0xa0 [osd_ldiskfs]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa17803fb>] ? iam_leaf_load+0x4b/0xd0 [osd_ldiskfs]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa1781268>] ? __iam_it_get+0x108/0x1d0 [osd_ldiskfs]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa178151e>] ? iam_it_get+0x2e/0x150 [osd_ldiskfs]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa177cc68>] ? __osd_oi_lookup+0x118/0x3b0 [osd_ldiskfs]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa177cf84>] ? osd_oi_lookup+0x84/0x150 [osd_ldiskfs]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa1797331>] ? osd_scrub_exec+0xa11/0x1ab0 [osd_ldiskfs]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa1794152>] ? osd_scrub_next+0x222/0x470 [osd_ldiskfs]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa1798899>] ? osd_inode_iteration+0x4c9/0xd50 [osd_ldiskfs]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa1796920>] ? osd_scrub_exec+0x0/0x1ab0 [osd_ldiskfs]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa1793f30>] ? osd_scrub_next+0x0/0x470 [osd_ldiskfs]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa179a245>] ? osd_scrub_main+0x875/0xed0 [osd_ldiskfs]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffffa17999d0>] ? osd_scrub_main+0x0/0xed0 [osd_ldiskfs]
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:15:17 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: ptlrpcd_14: page allocation failure. order:2, mode:0x20
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: Pid: 2678, comm: ptlrpcd_14 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: ptlrpcd_8: page allocation failure. order:2, mode:0x20
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: Pid: 2672, comm: ptlrpcd_8 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffffa1b846ec>] ? osp_attr_get_interpterer+0x6c/0xc0 [osp]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa1b9e77c>] ? osp_async_update_interpret+0x10c/0x290 [osp]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8a3fa>] ? after_reply+0xc9a/0xea0 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8e4fe>] ? ptlrpc_check_set+0x32e/0x1c60 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff812930c5>] ? rb_insert_color+0x125/0x160
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff812eaf5e>] ? intel_idle+0xde/0x170
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff812eaf41>] ? intel_idle+0xc1/0x170
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81426517>] ? cpuidle_idle_call+0xa7/0x140
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 10 12:19:02 cs04r-sc-mds03-01 kernel: [<ffffffff815236e7>] ? start_secondary+0x2be/0x301
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: ptlrpcd_0: page allocation failure. order:2, mode:0x20
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: ptlrpcd_10: page allocation failure. order:2, mode:0x20
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: Pid: 2674, comm: ptlrpcd_10 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0312a8a>] ? bnx2x_start_xmit+0x31a/0x1690 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: ptlrpcd_28: page allocation failure. order:2, mode:0x20
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: Pid: 2692, comm: ptlrpcd_28 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: Pid: 2664, comm: ptlrpcd_0 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:19:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: ptlrpcd_24: page allocation failure. order:2, mode:0x20
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: Pid: 2688, comm: ptlrpcd_24 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:20:08 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: ptlrpcd_6: page allocation failure. order:2, mode:0x20
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: Pid: 2670, comm: ptlrpcd_6 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:20:32 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:25:34 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c350009c0: rc = -2
Aug 10 12:25:34 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 260 previous similar messages
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: ptlrpcd_26: page allocation failure. order:2, mode:0x20
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: ptlrpcd_2: page allocation failure. order:2, mode:0x20
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: Pid: 2666, comm: ptlrpcd_2 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: Pid: 2690, comm: ptlrpcd_26 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:28:00 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: kswapd0: page allocation failure. order:2, mode:0x20
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: Pid: 340, comm: kswapd0 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8105858d>] ? mutex_spin_on_owner+0x8d/0xc0
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8152bcde>] ? __mutex_lock_slowpath+0x16e/0x210
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8152b72b>] ? mutex_lock+0x2b/0x50
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffffa0831013>] ? lu_cache_shrink+0x203/0x310 [obdclass]
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8113d8da>] ? shrink_slab+0x11a/0x1a0
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff81140c5a>] ? balance_pgdat+0x57a/0x800
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff81141014>] ? kswapd+0x134/0x3b0
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff81140ee0>] ? kswapd+0x0/0x3b0
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:28:04 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: ptlrpcd_12: page allocation failure. order:2, mode:0x20
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: Pid: 2676, comm: ptlrpcd_12 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff81138937>] ? get_dirty_limits+0x27/0x2f0
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:28:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: ptlrpcd_12: page allocation failure. order:2, mode:0x20
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: Pid: 2676, comm: ptlrpcd_12 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:28:55 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: ptlrpcd_26: page allocation failure. order:2, mode:0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: Pid: 2690, comm: ptlrpcd_26 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81064557>] ? load_balance_fair+0x267/0x300
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743a>] ? smp_call_function_many+0x1ea/0x260
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81064557>] ? load_balance_fair+0x267/0x300
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff81173b13>] ? cache_grow+0x93/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8117406a>] ? ____cache_alloc_node+0x8a/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174367>] ? fallback_alloc+0x227/0x270
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:28:57 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:29:32 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c3500b574: rc = -2
Aug 10 12:29:32 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 869 previous similar messages
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: ptlrpcd_24: page allocation failure. order:2, mode:0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: Pid: 2688, comm: ptlrpcd_24 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81138937>] ? get_dirty_limits+0x27/0x2f0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: ptlrpcd_24: page allocation failure. order:2, mode:0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: Pid: 2688, comm: ptlrpcd_24 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: ptlrpcd_24: page allocation failure. order:2, mode:0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: Pid: 2688, comm: ptlrpcd_24 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa03153c7>] ? bnx2x_rx_int+0xfc7/0x1670 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa03122a1>] ? bnx2x_msix_fp_int+0xd1/0x170 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:31:48 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: ptlrpcd_26: page allocation failure. order:2, mode:0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: Pid: 2690, comm: ptlrpcd_26 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: ptlrpcd_4: page allocation failure. order:2, mode:0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: Pid: 2668, comm: ptlrpcd_4 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: ptlrpcd_28: page allocation failure. order:2, mode:0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: Pid: 2692, comm: ptlrpcd_28 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81039787>] ? native_apic_msr_write+0x37/0x40
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff814b28c4>] ? tcp_transmit_skb+0x4b4/0x8b0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff810b743e>] ? smp_call_function_many+0x1ee/0x260
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81133ea0>] ? drain_local_pages+0x0/0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff810b74d2>] ? smp_call_function+0x22/0x30
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d594>] ? on_each_cpu+0x24/0x50
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81131d8c>] ? drain_all_pages+0x1c/0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113465d>] ? __alloc_pages_nodemask+0x5ed/0x8d0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:34:45 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: ptlrpcd_24: page allocation failure. order:2, mode:0x20
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: Pid: 2688, comm: ptlrpcd_24 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113ef99>] ? shrink_inactive_list+0x589/0x830
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:34:50 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: ptlrpcd_14: page allocation failure. order:2, mode:0x20
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: Pid: 2678, comm: ptlrpcd_14 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff814b28c4>] ? tcp_transmit_skb+0x4b4/0x8b0
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff814b6520>] ? tcp_keepalive_timer+0x0/0x280
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff81065519>] ? find_busiest_queue+0x69/0x150
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:37:11 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: ptlrpcd_4: page allocation failure. order:2, mode:0x20
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: Pid: 2668, comm: ptlrpcd_4 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8106d504>] ? enqueue_task_fair+0x64/0x100
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff810586cd>] ? check_preempt_curr+0x6d/0x90
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff810649fe>] ? try_to_wake_up+0x24e/0x3e0
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:37:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: ptlrpcd_4: page allocation failure. order:2, mode:0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Pid: 2668, comm: ptlrpcd_4 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: ptlrpcd_10: page allocation failure. order:2, mode:0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Pid: 2674, comm: ptlrpcd_10 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8152d731>] ? _spin_lock+0x21/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811c3dc5>] ? try_to_free_buffers+0x45/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0233d57>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa163c6f8>] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113d228>] ? __remove_mapping+0xb8/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811cb726>] ? blkdev_releasepage+0x36/0x50
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811239f0>] ? try_to_release_page+0x30/0x60
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e29f>] ? shrink_page_list.clone.3+0x5af/0x680
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: ptlrpcd_10: page allocation failure. order:2, mode:0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Pid: 2674, comm: ptlrpcd_10 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81064ba2>] ? default_wake_function+0x12/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff810577e9>] ? __wake_up_common+0x59/0x90
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8152d72e>] ? _spin_lock+0x1e/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811c3dc5>] ? try_to_free_buffers+0x45/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0233d57>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa163c6f8>] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113d228>] ? __remove_mapping+0xb8/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811cb726>] ? blkdev_releasepage+0x36/0x50
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811239f0>] ? try_to_release_page+0x30/0x60
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e29f>] ? shrink_page_list.clone.3+0x5af/0x680
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8152d731>] ? _spin_lock+0x21/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811c3dc5>] ? try_to_free_buffers+0x45/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0233d57>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa163c6f8>] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113d228>] ? __remove_mapping+0xb8/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811cb726>] ? blkdev_releasepage+0x36/0x50
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811239f0>] ? try_to_release_page+0x30/0x60
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e29f>] ? shrink_page_list.clone.3+0x5af/0x680
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: ptlrpcd_10: page allocation failure. order:2, mode:0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Pid: 2674, comm: ptlrpcd_10 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff814b28c4>] ? tcp_transmit_skb+0x4b4/0x8b0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8152d72e>] ? _spin_lock+0x1e/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811c3dc5>] ? try_to_free_buffers+0x45/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0233d57>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa163c6f8>] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113d228>] ? __remove_mapping+0xb8/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811cb726>] ? blkdev_releasepage+0x36/0x50
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811239f0>] ? try_to_release_page+0x30/0x60
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e29f>] ? shrink_page_list.clone.3+0x5af/0x680
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8152d72e>] ? _spin_lock+0x1e/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811c3dc5>] ? try_to_free_buffers+0x45/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0233d57>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa163c6f8>] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113d228>] ? __remove_mapping+0xb8/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811cb726>] ? blkdev_releasepage+0x36/0x50
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811239f0>] ? try_to_release_page+0x30/0x60
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e29f>] ? shrink_page_list.clone.3+0x5af/0x680
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8152d72e>] ? _spin_lock+0x1e/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811c3dc5>] ? try_to_free_buffers+0x45/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0233d57>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa163c6f8>] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113d228>] ? __remove_mapping+0xb8/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811cb726>] ? blkdev_releasepage+0x36/0x50
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811239f0>] ? try_to_release_page+0x30/0x60
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e29f>] ? shrink_page_list.clone.3+0x5af/0x680
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: ptlrpcd_20: page allocation failure. order:2, mode:0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Pid: 2684, comm: ptlrpcd_20 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8152d72e>] ? _spin_lock+0x1e/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811c3dc5>] ? try_to_free_buffers+0x45/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0233d57>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa163c6f8>] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113d228>] ? __remove_mapping+0xb8/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811cb726>] ? blkdev_releasepage+0x36/0x50
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811239f0>] ? try_to_release_page+0x30/0x60
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e29f>] ? shrink_page_list.clone.3+0x5af/0x680
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: mdt01_031: page allocation failure. order:2, mode:0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Pid: 3690, comm: mdt01_031 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8152d731>] ? _spin_lock+0x21/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811c425d>] ? __find_get_block_slow+0x5d/0x130
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811c4aa6>] ? __find_get_block+0xa6/0xe0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811c4b04>] ? __getblk+0x24/0x2a0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa1618135>] ? __ldiskfs_get_inode_loc+0xf5/0x3b0 [ldiskfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa161851c>] ? ldiskfs_get_inode_loc+0x1c/0x20 [ldiskfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa1647c2c>] ? ldiskfs_xattr_get+0x7c/0x330 [ldiskfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa08355c0>] ? lu_capainfo_get+0x20/0x30 [obdclass]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa164a03b>] ? ldiskfs_xattr_trusted_get+0x2b/0x30 [ldiskfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811b4577>] ? generic_getxattr+0x87/0x90
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa176d356>] ? osd_xattr_get+0x246/0x2f0 [osd_ldiskfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa1ac0e59>] ? lod_xattr_get+0x229/0x760 [lod]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa1b2f2b1>] ? mdd_xattr_get+0x111/0x400 [mdd]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa19d7094>] ? mdt_attr_get_complex+0x464/0xc00 [mdt]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa19fcc49>] ? mdt_reint_open+0xc59/0x2c70 [mdt]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa05ee83c>] ? upcall_cache_get_entry+0x29c/0x880 [libcfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa084e100>] ? lu_ucred+0x20/0x30 [obdclass]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa19c4ec5>] ? mdt_ucred+0x15/0x20 [mdt]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa19e51dd>] ? mdt_reint_rec+0x5d/0x200 [mdt]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa19c923b>] ? mdt_reint_internal+0x4cb/0x7a0 [mdt]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa19c9706>] ? mdt_intent_reint+0x1f6/0x430 [mdt]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa19c7cf4>] ? mdt_intent_policy+0x494/0xce0 [mdt]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e4f4f9>] ? ldlm_lock_enqueue+0x129/0x9d0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e7b46b>] ? ldlm_handle_enqueue0+0x51b/0x13f0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa05e3c8a>] ? lc_watchdog_touch+0x7a/0x190 [libcfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0efb921>] ? tgt_enqueue+0x61/0x230 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0efc56e>] ? tgt_request_handle+0x8be/0x1000 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0eac5a1>] ? ptlrpc_main+0xe41/0x1960 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8106c4f0>] ? pick_next_task_fair+0xd0/0x130
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0eab760>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff811c3d0f>] ? drop_buffers+0x4f/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811c3dd1>] ? try_to_free_buffers+0x51/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0233d57>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117ec51>] ? __mem_cgroup_uncharge_common+0x81/0x300
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa163c6f8>] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113d228>] ? __remove_mapping+0xb8/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811cb726>] ? blkdev_releasepage+0x36/0x50
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811239f0>] ? try_to_release_page+0x30/0x60
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e29f>] ? shrink_page_list.clone.3+0x5af/0x680
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117d62b>] ? mem_cgroup_lru_del_list+0x2b/0xb0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:37:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: __ratelimit: 207 callbacks suppressed
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: lfsck_layout: page allocation failure. order:2, mode:0x20
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: Pid: 38533, comm: lfsck_layout Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff814baa42>] ? tcp_v4_rcv+0x522/0x900
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81497a18>] ? ip_local_deliver+0x98/0xa0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa031460f>] ? bnx2x_rx_int+0x20f/0x1670 [bnx2x]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff812a3530>] ? swiotlb_map_page+0x0/0x100
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff81134b7a>] ? free_hot_cold_page+0x14a/0x220
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0233d57>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81134c92>] ? __pagevec_free+0x42/0x90
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113d228>] ? __remove_mapping+0xb8/0x160
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113e245>] ? shrink_page_list.clone.3+0x555/0x680
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e9b440>] ? lustre_pack_request+0xb0/0x170 [ptlrpc]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e9040e>] ? ptlrpc_set_wait+0x5de/0x890 [ptlrpc]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e9d750>] ? lustre_swab_object_update_request+0x0/0xb0 [ptlrpc]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e9adc6>] ? lustre_msg_set_jobid+0xb6/0x140 [ptlrpc]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e90741>] ? ptlrpc_queue_wait+0x81/0x220 [ptlrpc]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0f086d1>] ? out_remote_sync+0x111/0x200 [ptlrpc]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1b84a92>] ? osp_attr_get+0x352/0x600 [osp]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1951e50>] ? lfsck_layout_assistant_handler_p1+0x530/0x19f0 [lfsck]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa19186e6>] ? lfsck_assistant_engine+0x496/0x1de0 [lfsck]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1918250>] ? lfsck_assistant_engine+0x0/0x1de0 [lfsck]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: lfsck_layout: page allocation failure. order:2, mode:0x20
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: Pid: 38533, comm: lfsck_layout Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff814baa42>] ? tcp_v4_rcv+0x522/0x900
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81497a18>] ? ip_local_deliver+0x98/0xa0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa031460f>] ? bnx2x_rx_int+0x20f/0x1670 [bnx2x]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff812a3530>] ? swiotlb_map_page+0x0/0x100
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8152d722>] ? _spin_lock+0x12/0x30
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa176cb8a>] ? osd_attr_get+0xba/0x190 [osd_ldiskfs]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1951bf0>] ? lfsck_layout_assistant_handler_p1+0x2d0/0x19f0 [lfsck]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa19186e6>] ? lfsck_assistant_engine+0x496/0x1de0 [lfsck]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffffa1918250>] ? lfsck_assistant_engine+0x0/0x1de0 [lfsck]
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:43:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: lfsck_layout: page allocation failure. order:2, mode:0x20
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: Pid: 38533, comm: lfsck_layout Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff814baa42>] ? tcp_v4_rcv+0x522/0x900
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff81497a18>] ? ip_local_deliver+0x98/0xa0
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffffa031460f>] ? bnx2x_rx_int+0x20f/0x1670 [bnx2x]
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff8145d22b>] ? __netif_receive_skb+0x4ab/0x750
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff812a3530>] ? swiotlb_map_page+0x0/0x100
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffffa0831608>] ? lu_object_find_slice+0x48/0x80 [obdclass]
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffffa19519cb>] ? lfsck_layout_assistant_handler_p1+0xab/0x19f0 [lfsck]
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffffa19186e6>] ? lfsck_assistant_engine+0x496/0x1de0 [lfsck]
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffffa1918250>] ? lfsck_assistant_engine+0x0/0x1de0 [lfsck]
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:43:24 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 12:47:56 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200000405000047: rc = -2
Aug 10 12:47:56 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 11 previous similar messages
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8113e607>] ? isolate_lru_pages.clone.0+0xd7/0x170
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 12:59:39 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81133cd2>] ? free_pcppages_bulk+0x392/0x460
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff814b9133>] ? tcp_v4_do_rcv+0x2e3/0x490
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa03153c7>] ? bnx2x_rx_int+0xfc7/0x1670 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa03122a1>] ? bnx2x_msix_fp_int+0xd1/0x170 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: Pid: 2694, comm: ptlrpcd_30 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa03122a1>] ? bnx2x_msix_fp_int+0xd1/0x170 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0310182>] ? bnx2x_drain_tx_queues+0xd2/0x140 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff810b034a>] ? tick_program_event+0x2a/0x30
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff815344ca>] ? smp_apic_timer_interrupt+0x4a/0x60
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100bb93>] ? apic_timer_interrupt+0x13/0x20
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 13:24:23 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: ptlrpcd_4: page allocation failure. order:2, mode:0x20
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: Pid: 2668, comm: ptlrpcd_4 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 13:24:29 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: ptlrpcd_14: page allocation failure. order:2, mode:0x20
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: Pid: 2678, comm: ptlrpcd_14 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa03153c7>] ? bnx2x_rx_int+0xfc7/0x1670 [bnx2x]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8145086e>] ? __kfree_skb+0x1e/0xa0
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa03122a1>] ? bnx2x_msix_fp_int+0xd1/0x170 [bnx2x]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 13:24:34 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: ptlrpcd_22: page allocation failure. order:2, mode:0x20
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: Pid: 2686, comm: ptlrpcd_22 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff810149a9>] ? read_tsc+0x9/0x20
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: ptlrpcd_16: page allocation failure. order:2, mode:0x20
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: Pid: 2680, comm: ptlrpcd_16 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81450897>] ? __kfree_skb+0x47/0xa0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 13:26:21 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: ptlrpcd_18: page allocation failure. order:2, mode:0x20
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: Pid: 2682, comm: ptlrpcd_18 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: ptlrpcd_22: page allocation failure. order:2, mode:0x20
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: Pid: 2686, comm: ptlrpcd_22 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109828d>] ? insert_work+0x6d/0xb0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113e937>] ? shrink_active_list+0x297/0x370
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8152a27e>] ? thread_return+0x4e/0x7d0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbfbb>] ? ptlrpcd+0x35b/0x420 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: Call Trace:
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: <IRQ>  [<ffffffff811347ba>] ? __alloc_pages_nodemask+0x74a/0x8d0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa030fa27>] ? bnx2x_frag_alloc+0x17/0x20 [bnx2x]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314277>] ? bnx2x_alloc_rx_data+0x47/0x1d0 [bnx2x]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff812a2c88>] ? swiotlb_sync_single+0x28/0xd0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0314ea9>] ? bnx2x_rx_int+0xaa9/0x1670 [bnx2x]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81450a6a>] ? skb_release_head_state+0x6a/0x110
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa031250c>] ? bnx2x_free_tx_pkt+0x1cc/0x2e0 [bnx2x]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8145a2c3>] ? __napi_complete+0x23/0x40
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0315c8f>] ? bnx2x_poll+0x10f/0x400 [bnx2x]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81462a23>] ? net_rx_action+0x103/0x2f0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff810eaec0>] ? handle_IRQ_event+0x60/0x170
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100fb55>] ? do_softirq+0x65/0xa0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8107d765>] ? irq_exit+0x85/0x90
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81534405>] ? do_IRQ+0x75/0xf0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: <EOI>  [<ffffffff8113d220>] ? __remove_mapping+0xb0/0x160
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113e200>] ? shrink_page_list.clone.3+0x510/0x680
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113ed53>] ? shrink_inactive_list+0x343/0x830
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113887a>] ? determine_dirtyable_memory+0x1a/0x30
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa05e7897>] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113f5be>] ? shrink_mem_cgroup_zone+0x37e/0x540
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8117f2fd>] ? mem_cgroup_iter+0xfd/0x280
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113f7fa>] ? shrink_zone+0x7a/0x180
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8113fa15>] ? do_try_to_free_pages+0x115/0x610
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff811320bc>] ? get_page_from_freelist+0x15c/0x870
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff811400e2>] ? try_to_free_pages+0x92/0x120
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff811344ee>] ? __alloc_pages_nodemask+0x47e/0x8d0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff811736e2>] ? kmem_getpages+0x62/0x170
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff811742fa>] ? fallback_alloc+0x1ba/0x270
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81173d4f>] ? cache_grow+0x2cf/0x320
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174079>] ? ____cache_alloc_node+0x99/0x160
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81174cc9>] ? __kmalloc+0x199/0x230
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ed5f0a>] ? null_alloc_repbuf+0x6a/0x3b0 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ec4215>] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e969a1>] ? ptl_send_rpc+0x911/0xd40 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0eca634>] ? sptlrpc_req_refresh_ctx+0x154/0x910 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8aa23>] ? ptlrpc_send_new_req+0x423/0x770 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0e8ebb0>] ? ptlrpc_check_set+0x9e0/0x1c60 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebb993>] ? ptlrpcd_check+0x533/0x550 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbf2a>] ? ptlrpcd+0x2ca/0x420 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffffa0ebbc60>] ? ptlrpcd+0x0/0x420 [ptlrpc]
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Aug 10 13:26:22 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 10 13:42:45 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) srv-lustre03-MDT0000: Cannot find sequence 0x200004c35008f3d: rc = -2
Aug 10 13:42:45 cs04r-sc-mds03-01 kernel: LustreError: 38533:0:(fld_handler.c:260:fld_server_lookup()) Skipped 2 previous similar messages
Aug 10 14:16:42 cs04r-sc-mds03-01 kernel: Lustre: MGS: haven't heard from client c2462f25-c749-cb40-12a8-0e68281ea51d (at 172.23.138.66@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802d948c000, cur 1439212602 expire 1439212452 last 1439212375
Aug 10 14:16:42 cs04r-sc-mds03-01 kernel: Lustre: Skipped 1 previous similar message
Aug 10 15:19:25 cs04r-sc-mds03-01 kernel: Lustre: MGS: haven't heard from client 52bd2fae-1acc-15f1-caf4-829f489f75c6 (at 172.23.138.46@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802420f0000, cur 1439216365 expire 1439216215 last 1439216138
Aug 10 15:19:37 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000: haven't heard from client 6d1a031b-7de3-0f15-9538-d6c2113d2d3d (at 10.144.138.46@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882028231800, cur 1439216377 expire 1439216227 last 1439216150
