スクリプトは 2013年03月19日 20時03分04秒 に開始しました]0;ddnuser01@wk5:~/fromDDBJ[?1034h[ddnuser01@wk5 fromDDBJ]$ crash ./vmlinux ./vmcore crash 5.1.1-2.el6 Copyright (C) 2002-2011 Red Hat, Inc. Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. [?1034hGNU gdb (GDB) 7.0 Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... please wait... (gathering kmem slab cache data) please wait... (gathering module symbol data) please wait... (gathering task table data) please wait... (determining panic task) KERNEL: ./vmlinux DUMPFILE: ./vmcore [PARTIAL DUMP] CPUS: 16 DATE: Thu Mar 14 14:54:01 2013 UPTIME: 04:01:32 LOAD AVERAGE: 105.61, 102.70, 94.99 TASKS: 697 NODENAME: t324 RELEASE: 2.6.32-131.0.15.el6.x86_64 VERSION: #1 SMP Tue May 10 15:42:40 EDT 2011 MACHINE: x86_64 (2593 Mhz) MEMORY: 64 GB PANIC: "Oops: 0002 [#1] SMP " (check log for details) PID: 0 COMMAND: "swapper" TASK: ffff88081cb72080 (1 of 16) [THREAD_INFO: ffff88101c27a000] CPU: 3 STATE: TASK_RUNNING (SYSRQ) crash> crash> crash> bt [?1h= PID: 0 TASK: ffff88081cb72080 CPU: 3 COMMAND: "swapper" #0 [ffff88004ac639a0] machine_kexec at ffffffff810310db #1 [ffff88004ac63a00] crash_kexec at ffffffff810b63b2 #2 [ffff88004ac63ad0] oops_end at ffffffff814dec50 #3 [ffff88004ac63b00] no_context at ffffffff81040cdb #4 [ffff88004ac63b50] __bad_area_nosemaphore at ffffffff81040f65 #5 [ffff88004ac63ba0] bad_area_nosemaphore at ffffffff81041033 #6 [ffff88004ac63bb0] __do_page_fault at ffffffff8104170d #7 [ffff88004ac63cd0] do_page_fault at ffffffff814e0c3e #8 [ffff88004ac63d00] page_fault at ffffffff814ddfe5 [exception RIP: sysrq_handle_crash+22] RIP: ffffffff8131abc6 RSP: ffff88004ac63db8 RFLAGS: 00010096 RAX: 0000000000000010 RBX: 0000000000000063 RCX: 0000000000009d52 RDX: 0000000000000000 RSI: ffff8808175ad000 RDI: 0000000000000063 RBP: ffff88004ac63db8 R8: 0000000000000073 R9: 0000000000000001 R10: 0000000000000001 R11: 0000000000000000 R12: ffff8808175ad000 R13: ffffffff81a99e80 R14: 0000000000000082 R15: 0000000000000007 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #9 [ffff88004ac63dc0] __handle_sysrq at ffffffff8131ae82 #10 [ffff88004ac63e10] handle_sysrq at ffffffff8131af6b #11 [ffff88004ac63e20] serial8250_handle_port at ffffffff813351e8 #12 [ffff88004ac63e80] serial8250_interrupt at ffffffff8133537b #13 [ffff88004ac63ed0] handle_IRQ_event at ffffffff810d6940 #14 [ffff88004ac63f20] handle_edge_irq at ffffffff810d9036 #15 [ffff88004ac63f60] handle_irq at ffffffff8100df89 #16 [ffff88004ac63f80] do_IRQ at ffffffff814e34fc --- --- #17 [ffff88101c27be38] ret_from_intr at ffffffff8100bad3 [exception RIP: mwait_idle+119] RIP: ffffffff810141a7 RSP: ffff88101c27bee8 RFLAGS: 00000246 RAX: 0000000000000000 RBX: ffff88101c27bef8 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88101c27bfd8 RDI: ffff88081c8c6240 RBP: ffffffff8100bace R8: 0000000000000000 R9: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000000 R12: ffffffff81b7c0b8 R13: 0000000000000000 R14: ffffffff810ece03 R15: ffff88101c27be68 ORIG_RAX: ffffffffffffffcc CS: 0010 SS: 0018 #18 [ffff88101c27bf00] cpu_idle at ffffffff81009e96 [?1l>crash> crash> crash> runq [?1h= CPU 0 RUNQUEUE: ffff88004ac15f80 CURRENT: PID: 57858 TASK: ffff88100e6c1540 COMMAND: "sshd" RT PRIO_ARRAY: ffff88004ac16098 [no tasks queued] CFS RB_ROOT: ffff88004ac16010 [120] PID: 4862 TASK: ffff88100db7eb00 COMMAND: "xfslogd/0" [120] PID: 4878 TASK: ffff88100da9b500 COMMAND: "xfsdatad/0" [120] PID: 67 TASK: ffff88081c0c3500 COMMAND: "events/0" [120] PID: 106 TASK: ffff88081c347540 COMMAND: "kblockd/0" CPU 1 RUNQUEUE: ffff88004ac35f80 CURRENT: PID: 0 TASK: ffff88081cb2c0c0 COMMAND: "swapper" RT PRIO_ARRAY: ffff88004ac36098 [no tasks queued] CFS RB_ROOT: ffff88004ac36010 [no tasks queued] CPU 2 RUNQUEUE: ffff88004ac55f80 CURRENT: PID: 57737 TASK: ffff88100d4d4080 COMMAND: "bash" RT PRIO_ARRAY: ffff88004ac56098 [no tasks queued] CFS RB_ROOT: ffff88004ac56010 [120] PID: 69 TASK: ffff88081c0c2080 COMMAND: "events/2" CPU 3 RUNQUEUE: ffff88004ac75f80 CURRENT: PID: 0 TASK: ffff88081cb72080 COMMAND: "swapper" RT PRIO_ARRAY: ffff88004ac76098 [no tasks queued] CFS RB_ROOT: ffff88004ac76010 [120] PID: 70 TASK: ffff88081c0f3540 COMMAND: "events/3" CPU 4 RUNQUEUE: ffff88004ac95f80 CURRENT: PID: 0 TASK: ffff88101ca354c0 COMMAND: "swapper" RT PRIO_ARRAY: ffff88004ac96098 [no tasks queued] CFS RB_ROOT: ffff88004ac96010 [no tasks queued] CPU 5 RUNQUEUE: ffff88004acb5f80 CURRENT: PID: 0 TASK: ffff88081cbb6040 COMMAND: "swapper" RT PRIO_ARRAY: ffff88004acb6098 [no tasks queued] CFS RB_ROOT: ffff88004acb6010 [no tasks queued]  -- MORE -- forward: , or j backward: b or k quit: q CPU 6 RUNQUEUE: ffff88004acd5f80 CURRENT: PID: 181 TASK: ffff880817a9a0c0 COMMAND: "khugepaged" RT PRIO_ARRAY: ffff88004acd6098 [no tasks queued] CFS RB_ROOT: ffff88004acd6010 [120] PID: 73 TASK: ffff88081c0fd580 COMMAND: "events/6" [120] PID: 112 TASK: ffff88081c35b4c0 COMMAND: "kblockd/6" CPU 7 RUNQUEUE: ffff88004acf5f80 CURRENT: PID: 0 TASK: ffff88081cbfc100 COMMAND: "swapper" RT PRIO_ARRAY: ffff88004acf6098 [no tasks queued] CFS RB_ROOT: ffff88004acf6010 [no tasks queued] CPU 8 RUNQUEUE: ffff88085c415f80 CURRENT: PID: 57760 TASK: ffff880789dc00c0 COMMAND: "hpetfe" RT PRIO_ARRAY: ffff88085c416098 [no tasks queued] CFS RB_ROOT: ffff88085c416010 [120] PID: 57637 TASK: ffff88101712f580 COMMAND: "sleep" CPU 9 RUNQUEUE: ffff88085c435f80 CURRENT: PID: 31037 TASK: ffff881017a70080 COMMAND: "pg_basebackup" RT PRIO_ARRAY: ffff88085c436098 [no tasks queued] CFS RB_ROOT: ffff88085c436010 [120] PID: 76 TASK: ffff88081c1074c0 COMMAND: "events/9" [120] PID: 41 TASK: ffff88081c01eb00 COMMAND: "ksoftirqd/9" [120] PID: 4888 TASK: ffff88100d5380c0 COMMAND: "xfsdatad/9" CPU 10 RUNQUEUE: ffff88085c455f80 CURRENT: PID: 3703 TASK: ffff880816006b40 COMMAND: "cmahostd" RT PRIO_ARRAY: ffff88085c456098 [no tasks queued] CFS RB_ROOT: ffff88085c456010 [120] PID: 77 TASK: ffff88081c106a80 COMMAND: "events/10" [120] PID: 3759 TASK: ffff8807fb972040 COMMAND: "cmasm2d" CPU 11 RUNQUEUE: ffff88085c475f80 CURRENT: PID: 3232 TASK: ffff881016c2c100 COMMAND: "ll_ping" RT PRIO_ARRAY: ffff88085c476098 [no tasks queued] CFS RB_ROOT: ffff88085c476010 [120] PID: 78 TASK: ffff88081c106040 COMMAND: "events/11"  -- MORE -- forward: , or j backward: b or k quit: q  CPU 12 RUNQUEUE: ffff88085c495f80 CURRENT: PID: 57729 TASK: ffff88100d59a100 COMMAND: "sh" RT PRIO_ARRAY: ffff88085c496098 [no tasks queued] CFS RB_ROOT: ffff88085c496010 [120] PID: 79 TASK: ffff88081c111500 COMMAND: "events/12" CPU 13 RUNQUEUE: ffff88085c4b5f80 CURRENT: PID: 0 TASK: ffff88081c078040 COMMAND: "swapper" RT PRIO_ARRAY: ffff88085c4b6098 [no tasks queued] CFS RB_ROOT: ffff88085c4b6010 [no tasks queued] CPU 14 RUNQUEUE: ffff88085c4d5f80 CURRENT: PID: 3241 TASK: ffff881016c9e0c0 COMMAND: "ldlm_bl_00" RT PRIO_ARRAY: ffff88085c4d6098 [no tasks queued] CFS RB_ROOT: ffff88085c4d6010 [no tasks queued] CPU 15 RUNQUEUE: ffff88085c4f5f80 CURRENT: PID: 0 TASK: ffff88081c096100 COMMAND: "swapper" RT PRIO_ARRAY: ffff88085c4f6098 [no tasks queued] CFS RB_ROOT: ffff88085c4f6010 [no tasks queued] [?1l>crash> crash> crash> crash> bt all [?1h= bt: invalid task or pid value: all [?1l>crash> bt -a [?1h= PID: 57858 TASK: ffff88100e6c1540 CPU: 0 COMMAND: "sshd" #0 [ffff88004ac07e90] crash_nmi_callback at ffffffff81028a76 #1 [ffff88004ac07ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88004ac07ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88004ac07ef0] notify_die at ffffffff810940fe #4 [ffff88004ac07f20] do_nmi at ffffffff814de963 #5 [ffff88004ac07f50] nmi at ffffffff814de270 [exception RIP: smp_call_function_many+434] RIP: ffffffff810a4512 RSP: ffff8809a179b7d8 RFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff88004ac11160 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffff8809a179b818 R8: ffffffff81b9e088 R9: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000 R13: ffffffff81b9e080 R14: ffffffff81b9e080 R15: ffffffff8111fab0 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff8809a179b7d8] smp_call_function_many at ffffffff810a4512 #7 [ffff8809a179b820] smp_call_function at ffffffff810a4592 #8 [ffff8809a179b830] on_each_cpu at ffffffff8106f334 #9 [ffff8809a179b860] drain_all_pages at ffffffff8111e3ec #10 [ffff8809a179b870] __alloc_pages_nodemask at ffffffff8111ffdc #11 [ffff8809a179b990] kmem_getpages at ffffffff81159932 #12 [ffff8809a179b9c0] fallback_alloc at ffffffff8115a54a #13 [ffff8809a179ba40] ____cache_alloc_node at ffffffff8115a2c9 #14 [ffff8809a179baa0] kmem_cache_alloc_node_notrace at ffffffff8115b15f #15 [ffff8809a179bae0] __kmalloc_node at ffffffff8115b39b #16 [ffff8809a179bb30] __alloc_skb at ffffffff81414a4a #17 [ffff8809a179bb80] sock_alloc_send_pskb at ffffffff8140fee4 #18 [ffff8809a179bc20] sock_alloc_send_skb at ffffffff81410055 #19 [ffff8809a179bc30] unix_stream_sendmsg at ffffffff814ad0b0 #20 [ffff8809a179bcf0] sock_aio_write at ffffffff8140e601 #21 [ffff8809a179bdc0] do_sync_write at ffffffff8117241a #22 [ffff8809a179bef0] vfs_write at ffffffff811727e4 #23 [ffff8809a179bf30] sys_write at ffffffff81173151 #24 [ffff8809a179bf80] tracesys at ffffffff8100b387 (via system_call) RIP: 00007f04f4a715e0 RSP: 00007fff0a9bde08 RFLAGS: 00000246 RAX: ffffffffffffffda RBX: ffffffff8100b387 RCX: ffffffffffffffff RDX: 0000000000004e05 RSI: 00007f04f89fe990 RDI: 0000000000000006 RBP: 0000000000000006 R8: 0000000000000000 R9: 00007f04f769f9be R10: 00007f04f769f9a8 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f04f89fe990 R14: 00007f04f4a715d0 R15: 00007fff0a9bde20 ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b PID: 0 TASK: ffff88081cb2c0c0 CPU: 1 COMMAND: "swapper" #0 [ffff88004ac27e90] crash_nmi_callback at ffffffff81028a76  -- MORE -- forward: , or j backward: b or k quit: q  #1 [ffff88004ac27ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88004ac27ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88004ac27ef0] notify_die at ffffffff810940fe #4 [ffff88004ac27f20] do_nmi at ffffffff814de963 #5 [ffff88004ac27f50] nmi at ffffffff814de270 [exception RIP: mwait_idle+119] RIP: ffffffff810141a7 RSP: ffff88101cad7ee8 RFLAGS: 00000246 RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88101cad7fd8 RDI: ffff88081c8c6240 RBP: ffff88101cad7ef8 R8: 0000000000000000 R9: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000000 R12: ffff8807e8c0a758 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88101cad7ee8] mwait_idle at ffffffff810141a7 #7 [ffff88101cad7f00] cpu_idle at ffffffff81009e96 PID: 57737 TASK: ffff88100d4d4080 CPU: 2 COMMAND: "bash" #0 [ffff88004ac47e90] crash_nmi_callback at ffffffff81028a76 #1 [ffff88004ac47ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88004ac47ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88004ac47ef0] notify_die at ffffffff810940fe #4 [ffff88004ac47f20] do_nmi at ffffffff814de963 #5 [ffff88004ac47f50] nmi at ffffffff814de270 [exception RIP: _spin_lock+30] RIP: ffffffff814ddade RSP: ffff880ef3a875c8 RFLAGS: 00000293 RAX: 00000000000066de RBX: ffff8805642bf400 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffff880ef3a875c8 R8: 0000000000000000 R9: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000001 R13: ffff8805642bf400 R14: ffff8807ff5b80e0 R15: ffff8807ff5b8000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff880ef3a875c8] _spin_lock at ffffffff814ddade #7 [ffff880ef3a875d0] ptlrpc_queue_wait at ffffffffa0637cd8 [ptlrpc] #8 [ffff880ef3a87710] ldlm_cli_enqueue at ffffffffa061c028 [ptlrpc] #9 [ffff880ef3a877c0] mdc_enqueue at ffffffffa07882a5 [mdc] #10 [ffff880ef3a878d0] mdc_intent_lock at ffffffffa07897d2 [mdc] #11 [ffff880ef3a879d0] ll_lookup_it at ffffffffa08480b5 [lustre] #12 [ffff880ef3a87b40] ll_lookup_nd at ffffffffa0848bff [lustre] #13 [ffff880ef3a87ba0] do_lookup at ffffffff8118067b #14 [ffff880ef3a87c00] __link_path_walk at ffffffff811808e3 #15 [ffff880ef3a87c80] do_follow_link at ffffffff811810c8 #16 [ffff880ef3a87ce0] __link_path_walk at ffffffff8118091b #17 [ffff880ef3a87d60] path_walk at ffffffff811815fa  -- MORE -- forward: , or j backward: b or k quit: q #18 [ffff880ef3a87da0] do_path_lookup at ffffffff811817cb #19 [ffff880ef3a87dd0] user_path_at at ffffffff81181957 #20 [ffff880ef3a87ea0] vfs_fstatat at ffffffff81177c7c #21 [ffff880ef3a87ee0] vfs_stat at ffffffff81177deb #22 [ffff880ef3a87ef0] sys_newstat at ffffffff81177e14 #23 [ffff880ef3a87f80] tracesys at ffffffff8100b387 (via system_call) RIP: 00000031470d8eb5 RSP: 00007fffd2720a78 RFLAGS: 00000246 RAX: ffffffffffffffda RBX: ffffffff8100b387 RCX: ffffffffffffffff RDX: 00007fffd2720a80 RSI: 00007fffd2720a80 RDI: 0000000002200b70 RBP: 0000000002200b70 R8: 0000000000000001 R9: 0000000000000005 R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000001 R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: 0000000000000004 CS: 0033 SS: 002b PID: 0 TASK: ffff88081cb72080 CPU: 3 COMMAND: "swapper" #0 [ffff88004ac639a0] machine_kexec at ffffffff810310db #1 [ffff88004ac63a00] crash_kexec at ffffffff810b63b2 #2 [ffff88004ac63ad0] oops_end at ffffffff814dec50 #3 [ffff88004ac63b00] no_context at ffffffff81040cdb #4 [ffff88004ac63b50] __bad_area_nosemaphore at ffffffff81040f65 #5 [ffff88004ac63ba0] bad_area_nosemaphore at ffffffff81041033 #6 [ffff88004ac63bb0] __do_page_fault at ffffffff8104170d #7 [ffff88004ac63cd0] do_page_fault at ffffffff814e0c3e #8 [ffff88004ac63d00] page_fault at ffffffff814ddfe5 [exception RIP: sysrq_handle_crash+22] RIP: ffffffff8131abc6 RSP: ffff88004ac63db8 RFLAGS: 00010096 RAX: 0000000000000010 RBX: 0000000000000063 RCX: 0000000000009d52 RDX: 0000000000000000 RSI: ffff8808175ad000 RDI: 0000000000000063 RBP: ffff88004ac63db8 R8: 0000000000000073 R9: 0000000000000001 R10: 0000000000000001 R11: 0000000000000000 R12: ffff8808175ad000 R13: ffffffff81a99e80 R14: 0000000000000082 R15: 0000000000000007 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #9 [ffff88004ac63dc0] __handle_sysrq at ffffffff8131ae82 #10 [ffff88004ac63e10] handle_sysrq at ffffffff8131af6b #11 [ffff88004ac63e20] serial8250_handle_port at ffffffff813351e8 #12 [ffff88004ac63e80] serial8250_interrupt at ffffffff8133537b #13 [ffff88004ac63ed0] handle_IRQ_event at ffffffff810d6940 #14 [ffff88004ac63f20] handle_edge_irq at ffffffff810d9036 #15 [ffff88004ac63f60] handle_irq at ffffffff8100df89 #16 [ffff88004ac63f80] do_IRQ at ffffffff814e34fc --- --- #17 [ffff88101c27be38] ret_from_intr at ffffffff8100bad3 [exception RIP: mwait_idle+119] RIP: ffffffff810141a7 RSP: ffff88101c27bee8 RFLAGS: 00000246 RAX: 0000000000000000 RBX: ffff88101c27bef8 RCX: 0000000000000000  -- MORE -- forward: , or j backward: b or k quit: q  RDX: 0000000000000000 RSI: ffff88101c27bfd8 RDI: ffff88081c8c6240 RBP: ffffffff8100bace R8: 0000000000000000 R9: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000000 R12: ffffffff81b7c0b8 R13: 0000000000000000 R14: ffffffff810ece03 R15: ffff88101c27be68 ORIG_RAX: ffffffffffffffcc CS: 0010 SS: 0018 #18 [ffff88101c27bf00] cpu_idle at ffffffff81009e96 PID: 0 TASK: ffff88101ca354c0 CPU: 4 COMMAND: "swapper" #0 [ffff88004ac87e90] crash_nmi_callback at ffffffff81028a76 #1 [ffff88004ac87ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88004ac87ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88004ac87ef0] notify_die at ffffffff810940fe #4 [ffff88004ac87f20] do_nmi at ffffffff814de963 #5 [ffff88004ac87f50] nmi at ffffffff814de270 [exception RIP: mwait_idle+119] RIP: ffffffff810141a7 RSP: ffff88081cbafee8 RFLAGS: 00000246 RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88081cbaffd8 RDI: ffff88081c8c6240 RBP: ffff88081cbafef8 R8: 0000000000000000 R9: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000000 R12: ffff8807e8c0a758 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88081cbafee8] mwait_idle at ffffffff810141a7 #7 [ffff88081cbaff00] cpu_idle at ffffffff81009e96 PID: 0 TASK: ffff88081cbb6040 CPU: 5 COMMAND: "swapper" #0 [ffff88004aca7e90] crash_nmi_callback at ffffffff81028a76 #1 [ffff88004aca7ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88004aca7ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88004aca7ef0] notify_die at ffffffff810940fe #4 [ffff88004aca7f20] do_nmi at ffffffff814de963 #5 [ffff88004aca7f50] nmi at ffffffff814de270 [exception RIP: mwait_idle+119] RIP: ffffffff810141a7 RSP: ffff88101c27dee8 RFLAGS: 00000246 RAX: 0000000000000000 RBX: 0000000000000005 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88101c27dfd8 RDI: ffff88081c8c6240 RBP: ffff88101c27def8 R8: 0000000000000000 R9: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000000 R12: ffff8807e8c0a758 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88101c27dee8] mwait_idle at ffffffff810141a7 #7 [ffff88101c27df00] cpu_idle at ffffffff81009e96  -- MORE -- forward: , or j backward: b or k quit: q PID: 181 TASK: ffff880817a9a0c0 CPU: 6 COMMAND: "khugepaged" #0 [ffff88004acc7e90] crash_nmi_callback at ffffffff81028a76 #1 [ffff88004acc7ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88004acc7ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88004acc7ef0] notify_die at ffffffff810940fe #4 [ffff88004acc7f20] do_nmi at ffffffff814de963 #5 [ffff88004acc7f50] nmi at ffffffff814de270 [exception RIP: smp_call_function_many+434] RIP: ffffffff810a4512 RSP: ffff880817aa3b90 RFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff88004acd1160 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffff880817aa3bd0 R8: ffffffff81b9e088 R9: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000006 R13: ffffffff81b9e080 R14: ffffffff81b9e080 R15: ffffffff8111fab0 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff880817aa3b90] smp_call_function_many at ffffffff810a4512 #7 [ffff880817aa3bd8] smp_call_function at ffffffff810a4592 #8 [ffff880817aa3be8] on_each_cpu at ffffffff8106f334 #9 [ffff880817aa3c18] drain_all_pages at ffffffff8111e3ec #10 [ffff880817aa3c28] __alloc_pages_nodemask at ffffffff8111ffdc #11 [ffff880817aa3d48] alloc_pages_vma at ffffffff8115474a #12 [ffff880817aa3d98] khugepaged at ffffffff8116ad29 #13 [ffff880817aa3ee8] kthread at ffffffff8108ddf6 #14 [ffff880817aa3f48] kernel_thread at ffffffff8100c1ca PID: 0 TASK: ffff88081cbfc100 CPU: 7 COMMAND: "swapper" #0 [ffff88004ace7e90] crash_nmi_callback at ffffffff81028a76 #1 [ffff88004ace7ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88004ace7ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88004ace7ef0] notify_die at ffffffff810940fe #4 [ffff88004ace7f20] do_nmi at ffffffff814de963 #5 [ffff88004ace7f50] nmi at ffffffff814de270 [exception RIP: mwait_idle+119] RIP: ffffffff810141a7 RSP: ffff88101c281ee8 RFLAGS: 00000246 RAX: 0000000000000000 RBX: 0000000000000007 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88101c281fd8 RDI: ffff88081c8c6240 RBP: ffff88101c281ef8 R8: 0000000000000000 R9: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000000 R12: ffff8807e8c0a758 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88101c281ee8] mwait_idle at ffffffff810141a7 #7 [ffff88101c281f00] cpu_idle at ffffffff81009e96  -- MORE -- forward: , or j backward: b or k quit: q PID: 57760 TASK: ffff880789dc00c0 CPU: 8 COMMAND: "hpetfe" #0 [ffff88085c407e90] crash_nmi_callback at ffffffff81028a76 #1 [ffff88085c407ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88085c407ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88085c407ef0] notify_die at ffffffff810940fe #4 [ffff88085c407f20] do_nmi at ffffffff814de963 #5 [ffff88085c407f50] nmi at ffffffff814de270 [exception RIP: int_signal+13] RIP: ffffffff8100b43b RSP: ffff88009ede5f58 RFLAGS: 00000246 RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88009ede4000 RDX: 0000000000000800 RSI: 0000000000000000 RDI: ffff88009ede5f58 RBP: 00000000ffffffff R8: ffff88009ede4000 R9: 0000000000000001 R10: 00000000ffffffff R11: 0000000000000000 R12: 000000000000e100 R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88009ede5f58] int_signal at ffffffff8100b43b RIP: 00000031470ab64e RSP: 00007fff8fcbd060 RFLAGS: 00000246 RAX: 000000000000e1a1 RBX: 0000000000000000 RCX: ffffffffffffffff RDX: 0000000000000000 RSI: 00007fff8fcbd0cc RDI: ffffffffffffffff RBP: 00000000ffffffff R8: 0000000001f36250 R9: 0000000001f6b8a0 R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000e100 R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: 000000000000003d CS: 0033 SS: 002b PID: 31037 TASK: ffff881017a70080 CPU: 9 COMMAND: "pg_basebackup" #0 [ffff88085c427e90] crash_nmi_callback at ffffffff81028a76 #1 [ffff88085c427ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88085c427ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88085c427ef0] notify_die at ffffffff810940fe #4 [ffff88085c427f20] do_nmi at ffffffff814de963 #5 [ffff88085c427f50] nmi at ffffffff814de270 [exception RIP: smp_call_function_many+438] RIP: ffffffff810a4516 RSP: ffff8808685ad5d8 RFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff88085c431160 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffff8808685ad618 R8: ffffffff81b9e088 R9: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000009 R13: ffffffff81b9e080 R14: ffffffff81b9e080 R15: ffffffff8111fab0 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff8808685ad5d8] smp_call_function_many at ffffffff810a4516 #7 [ffff8808685ad620] smp_call_function at ffffffff810a4592 #8 [ffff8808685ad630] on_each_cpu at ffffffff8106f334 #9 [ffff8808685ad660] drain_all_pages at ffffffff8111e3ec  -- MORE -- forward: , or j backward: b or k quit: q #10 [ffff8808685ad670] __alloc_pages_nodemask at ffffffff8111ffdc #11 [ffff8808685ad790] alloc_pages_vma at ffffffff8115474a #12 [ffff8808685ad7e0] do_huge_pmd_anonymous_page at ffffffff8116d2ee #13 [ffff8808685ad840] handle_mm_fault at ffffffff8113836a #14 [ffff8808685ad890] __do_page_fault at ffffffff81041529 #15 [ffff8808685ad9b0] do_page_fault at ffffffff814e0c3e #16 [ffff8808685ad9e0] page_fault at ffffffff814ddfe5 [exception RIP: copy_user_generic_string+45] RIP: ffffffff8126d2ad RSP: ffff8808685ada90 RFLAGS: 00010206 RAX: ffff8808685ac075 RBX: ffff8808685ade98 RCX: 00000000000000e5 RDX: 0000000000000003 RSI: ffff880ea5cbe8f5 RDI: 00007ff874400000 RBP: ffff8808685adab8 R8: 000000008d494126 R9: 0000000000000000 R10: ffff880800aca180 R11: 000000000000ffc8 R12: 0000000000000fc8 R13: ffff880ea5cbe058 R14: 0000000000000fc8 R15: 0000000000000fc8 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #17 [ffff8808685ada90] memcpy_toiovec at ffffffff814178d5 #18 [ffff8808685adac0] skb_copy_datagram_iovec at ffffffff814182f0 #19 [ffff8808685adb30] tcp_recvmsg at ffffffff81462685 #20 [ffff8808685adc40] inet_recvmsg at ffffffff81482a7c #21 [ffff8808685adc80] sock_recvmsg at ffffffff8140c5e3 #22 [ffff8808685ade40] sys_recvfrom at ffffffff8140c93e #23 [ffff8808685adf80] tracesys at ffffffff8100b387 (via system_call) RIP: 00000031470e7652 RSP: 00007fff48457bb0 RFLAGS: 00000246 RAX: ffffffffffffffda RBX: ffffffff8100b387 RCX: ffffffffffffffff RDX: 0000000002250fed RSI: 00007ff874180023 RDI: 0000000000000004 RBP: 0000000000000001 R8: 0000000000000000 R9: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff484585a0 R13: 00000000020ac4b0 R14: 00007fff48458e28 R15: 0000000000000000 ORIG_RAX: 000000000000002d CS: 0033 SS: 002b PID: 3703 TASK: ffff880816006b40 CPU: 10 COMMAND: "cmahostd" #0 [ffff88085c447e90] crash_nmi_callback at ffffffff81028a76 #1 [ffff88085c447ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88085c447ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88085c447ef0] notify_die at ffffffff810940fe #4 [ffff88085c447f20] do_nmi at ffffffff814de963 #5 [ffff88085c447f50] nmi at ffffffff814de270 [exception RIP: _spin_lock+28] RIP: ffffffff814ddadc RSP: ffff8807fadc39d8 RFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff88101c2e9000 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffff8807fadc39d8 R8: 0000000000000004 R9: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: 0000000000000000 R14: ffff8807ff5b80e0 R15: ffff8807ff5b8000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018  -- MORE -- forward: , or j backward: b or k quit: q --- --- #6 [ffff8807fadc39d8] _spin_lock at ffffffff814ddadc #7 [ffff8807fadc39e0] ptlrpc_queue_wait at ffffffffa0637cd8 [ptlrpc] #8 [ffff8807fadc3b20] mdc_statfs at ffffffffa077a471 [mdc] #9 [ffff8807fadc3bb0] ll_statfs_internal at ffffffffa082355c [lustre] #10 [ffff8807fadc3d00] ll_statfs at ffffffffa0823e67 [lustre] #11 [ffff8807fadc3e00] vfs_statfs at ffffffff81170cdc #12 [ffff8807fadc3e20] vfs_statfs_native at ffffffff81170f30 #13 [ffff8807fadc3ec0] sys_statfs at ffffffff811710c3 #14 [ffff8807fadc3f80] tracesys at ffffffff8100b387 (via system_call) RIP: 00000031470d90a7 RSP: 00007fff78a7e198 RFLAGS: 00000206 RAX: ffffffffffffffda RBX: ffffffff8100b387 RCX: ffffffffffffffff RDX: 00000000012e5b05 RSI: 00007fff78a7e230 RDI: 00000000012e9d30 RBP: 00007fff78a7e230 R8: 0000000000001ff6 R9: 0000000040000000 R10: fffffffffffffff0 R11: 0000000000000206 R12: 00000000012e5b05 R13: 00007fff78a80650 R14: 00007fff78a7e4e0 R15: 00000000012e9d30 ORIG_RAX: 0000000000000089 CS: 0033 SS: 002b PID: 3232 TASK: ffff881016c2c100 CPU: 11 COMMAND: "ll_ping" #0 [ffff88085c467e90] crash_nmi_callback at ffffffff81028a76 #1 [ffff88085c467ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88085c467ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88085c467ef0] notify_die at ffffffff810940fe #4 [ffff88085c467f20] do_nmi at ffffffff814de963 #5 [ffff88085c467f50] nmi at ffffffff814de270 [exception RIP: _spin_lock+30] RIP: ffffffff814ddade RSP: ffff881018e45e40 RFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff8807ff5b8000 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffff881018e45e40 R8: 0000000000000000 R9: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff8807ff5b8070 R13: 0000000000000009 R14: 0000000000000000 R15: 0000000100acec60 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff881018e45e40] _spin_lock at ffffffff814ddade #7 [ffff881018e45e48] ptlrpc_pinger_main at ffffffffa065537f [ptlrpc] #8 [ffff881018e45f48] kernel_thread at ffffffff8100c1ca PID: 57729 TASK: ffff88100d59a100 CPU: 12 COMMAND: "sh" #0 [ffff88085c487e90] crash_nmi_callback at ffffffff81028a76 #1 [ffff88085c487ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88085c487ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88085c487ef0] notify_die at ffffffff810940fe #4 [ffff88085c487f20] do_nmi at ffffffff814de963 #5 [ffff88085c487f50] nmi at ffffffff814de270  -- MORE -- forward: , or j backward: b or k quit: q  [exception RIP: smp_call_function_many+438] RIP: ffffffff810a4516 RSP: ffff88009b87dbf0 RFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88085c491160 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffff88009b87dc30 R8: ffffffff81b9e088 R9: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 000000000000000c R13: ffffffff81b9e080 R14: ffffffff81b9e080 R15: ffffffff8111fab0 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88009b87dbf0] smp_call_function_many at ffffffff810a4516 #7 [ffff88009b87dc38] smp_call_function at ffffffff810a4592 #8 [ffff88009b87dc48] on_each_cpu at ffffffff8106f334 #9 [ffff88009b87dc78] drain_all_pages at ffffffff8111e3ec #10 [ffff88009b87dc88] __alloc_pages_nodemask at ffffffff8111ffdc #11 [ffff88009b87dda8] alloc_pages_current at ffffffff8115464a #12 [ffff88009b87ddd8] __get_free_pages at ffffffff8111e36e #13 [ffff88009b87dde8] copy_process at ffffffff81064f14 #14 [ffff88009b87de78] do_fork at ffffffff810661c4 #15 [ffff88009b87df38] sys_clone at ffffffff81009588 #16 [ffff88009b87df48] stub_clone at ffffffff8100b493 RIP: 00000031470abb4d RSP: 00007fff00b83bb0 RFLAGS: 00000246 RAX: ffffffffffffffda RBX: 00007fff00b83bb0 RCX: ffffffffffffffff RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011 RBP: 00007fff00b83bf0 R8: 0000000000000000 R9: 0000000000000000 R10: 00007ffdc1dbf9d0 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000000 R14: 00000000014c1380 R15: 00000000014c0b60 ORIG_RAX: 0000000000000038 CS: 0033 SS: 002b PID: 0 TASK: ffff88081c078040 CPU: 13 COMMAND: "swapper" #0 [ffff88085c4a7e90] crash_nmi_callback at ffffffff81028a76 #1 [ffff88085c4a7ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88085c4a7ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88085c4a7ef0] notify_die at ffffffff810940fe #4 [ffff88085c4a7f20] do_nmi at ffffffff814de963 #5 [ffff88085c4a7f50] nmi at ffffffff814de270 [exception RIP: mwait_idle+119] RIP: ffffffff810141a7 RSP: ffff88101c3bbee8 RFLAGS: 00000246 RAX: 0000000000000000 RBX: 000000000000000d RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88101c3bbfd8 RDI: ffff88081c8c6240 RBP: ffff88101c3bbef8 R8: 0000000000000000 R9: 000000003c200000 R10: ffff8807e8ddec02 R11: ffff8807e8ddec41 R12: ffff8807e8c0a758 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88101c3bbee8] mwait_idle at ffffffff810141a7  -- MORE -- forward: , or j backward: b or k quit: q  #7 [ffff88101c3bbf00] cpu_idle at ffffffff81009e96 PID: 3241 TASK: ffff881016c9e0c0 CPU: 14 COMMAND: "ldlm_bl_00" #0 [ffff88085c4c7e90] crash_nmi_callback at ffffffff81028a76 #1 [ffff88085c4c7ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88085c4c7ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88085c4c7ef0] notify_die at ffffffff810940fe #4 [ffff88085c4c7f20] do_nmi at ffffffff814de963 #5 [ffff88085c4c7f50] nmi at ffffffff814de270 [exception RIP: _spin_lock+30] RIP: ffffffff814ddade RSP: ffff8810113a35c0 RFLAGS: 00000097 RAX: 0000000000004ac4 RBX: 0000000000000158 RCX: 0000000000000010 RDX: 0000000000004ac3 RSI: 0000000000000000 RDI: ffffffff81f2cdb8 RBP: ffff8810113a35c0 R8: ffffffff81b9e288 R9: 0000000000000040 R10: ffff88086b93d1c2 R11: 0000000000000000 R12: ffffffff81f2cdb8 R13: 0000000000000001 R14: 0000000000000042 R15: ffffffff81d70c23 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff8810113a35c0] _spin_lock at ffffffff814ddade #7 [ffff8810113a35c8] serial8250_console_write at ffffffff813326ce #8 [ffff8810113a3618] __call_console_drivers at ffffffff810672d5 #9 [ffff8810113a3648] _call_console_drivers at ffffffff8106733a #10 [ffff8810113a3668] release_console_sem at ffffffff81067908 #11 [ffff8810113a36a8] vprintk at ffffffff81067f88 #12 [ffff8810113a3748] printk at ffffffff814dad34 #13 [ffff8810113a37a8] print_to_console at ffffffffa04bc194 [libcfs] #14 [ffff8810113a37d8] libcfs_debug_vmsg2 at ffffffffa04c50c5 [libcfs] #15 [ffff8810113a3918] ptlrpc_set_import_discon at ffffffffa066bc02 [ptlrpc] #16 [ffff8810113a39a8] ptlrpc_request_handle_notconn at ffffffffa063961b [ptlrpc] #17 [ffff8810113a39f8] after_reply at ffffffffa06319a4 [ptlrpc] #18 [ffff8810113a3a98] ptlrpc_queue_wait at ffffffffa0638486 [ptlrpc] #19 [ffff8810113a3bd8] ldlm_cli_cancel_req at ffffffffa0618edc [ptlrpc] #20 [ffff8810113a3c78] ldlm_cli_cancel_list at ffffffffa061966a [ptlrpc] #21 [ffff8810113a3cf8] ldlm_cli_cancel at ffffffffa061b1dc [ptlrpc] #22 [ffff8810113a3d88] ll_mdc_blocking_ast at ffffffffa0845677 [lustre] #23 [ffff8810113a3e18] ldlm_handle_bl_callback at ffffffffa061f11b [ptlrpc] #24 [ffff8810113a3e78] ldlm_bl_thread_main at ffffffffa0627f71 [ptlrpc] #25 [ffff8810113a3f48] kernel_thread at ffffffff8100c1ca PID: 0 TASK: ffff88081c096100 CPU: 15 COMMAND: "swapper" #0 [ffff88085c4e7e90] crash_nmi_callback at ffffffff81028a76 #1 [ffff88085c4e7ea0] notifier_call_chain at ffffffff814e0cf5 #2 [ffff88085c4e7ee0] atomic_notifier_call_chain at ffffffff814e0d5a #3 [ffff88085c4e7ef0] notify_die at ffffffff810940fe #4 [ffff88085c4e7f20] do_nmi at ffffffff814de963  -- MORE -- forward: , or j backward: b or k quit: q  #5 [ffff88085c4e7f50] nmi at ffffffff814de270 [exception RIP: mwait_idle+119] RIP: ffffffff810141a7 RSP: ffff88101c3e5ee8 RFLAGS: 00000246 RAX: 0000000000000000 RBX: 000000000000000f RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88101c3e5fd8 RDI: ffff88081c8c6240 RBP: ffff88101c3e5ef8 R8: 0000000000000000 R9: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000000 R12: ffff8807e8c0a758 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88101c3e5ee8] mwait_idle at ffffffff810141a7 #7 [ffff88101c3e5f00] cpu_idle at ffffffff81009e96 [?1l>crash> crash> crash> help bt [?1h= NAME bt - backtrace SYNOPSIS bt [-a|-g|-r|-t|-T|-l|-e|-E|-f|-F|-o|-O] [-R ref] [-I ip] [-S sp] [pid | task] DESCRIPTION Display a kernel stack backtrace. If no arguments are given, the stack trace of the current context will be displayed. -a displays the stack traces of the active task on each CPU. (only applicable to crash dumps) -g displays the stack traces of all threads in the thread group of the target task; the thread group leader will be displayed first. -r display raw stack data, consisting of a memory dump of the two pages of memory containing the task_union structure. -t display all text symbols found from the last known stack location to the top of the stack. (helpful if the back trace fails) -T display all text symbols found from just above the task_struct or thread_info to the top of the stack. (helpful if the back trace fails or the -t option starts too high in the process stack). -l show file and line number of each stack trace text location. -e search the stack for possible kernel and user mode exception frames. -E search the IRQ stacks (x86, x86_64 and ppc64), and the exception stacks (x86_64) for possible exception frames; all other arguments will be ignored since this is not a context-sensitive operation. -f display all stack data contained in a frame; this option can be used to determine the arguments passed to each function; on ia64, the argument register contents are dumped. -F similar to -f, except that the stack data is displayed symbolically when appropriate; if the stack data references a slab cache object, the name of the slab cache will be displayed in brackets; on ia64, the substitution is done to the argument register contents. -o x86: use old backtrace method, permissable only on kernels that were compiled without the -fomit-frame_pointer. x86_64: use old backtrace method, which dumps potentially stale kernel text return addresses found on the stack. -O x86: use old backtrace method by default, permissable only on kernels that were compiled without the -fomit-frame_pointer; subsequent usage of this option toggles the backtrace method. x86_64: use old backtrace method by default; subsequent usage of this option toggles the backtrace method. -R ref display stack trace only if there is a reference to this symbol or text address.  -- MORE -- forward: , or j backward: b or k quit: q [?1l>crash> crash> crash> crash> log [?1h= SQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U ) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3 (U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpi lo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcache jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0  -- MORE -- forward: , or j backward: b or k quit: q  [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib]  -- MORE -- forward: , or j backward: b or k quit: q  [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040  -- MORE -- forward: , or j backward: b or k quit: q R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U  -- MORE -- forward: , or j backward: b or k quit: q ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan]  -- MORE -- forward: , or j backward: b or k quit: q Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000  -- MORE -- forward: , or j backward: b or k quit: q DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000  -- MORE -- forward: , or j backward: b or k quit: q DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065  -- MORE -- forward: , or j backward: b or k quit: q R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U  -- MORE -- forward: , or j backward: b or k quit: q ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0  -- MORE -- forward: , or j backward: b or k quit: q  [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20  -- MORE -- forward: , or j backward: b or k quit: q  [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b  -- MORE -- forward: , or j backward: b or k quit: q CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400  -- MORE -- forward: , or j backward: b or k quit: q Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b0/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370  -- MORE -- forward: , or j backward: b or k quit: q  [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110  -- MORE -- forward: , or j backward: b or k quit: q  [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc]  -- MORE -- forward: , or j backward: b or k quit: q  [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202  -- MORE -- forward: , or j backward: b or k quit: q RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858]  -- MORE -- forward: , or j backward: b or k quit: q Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0  -- MORE -- forward: , or j backward: b or k quit: q  [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40  -- MORE -- forward: , or j backward: b or k quit: q  [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1c/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan]  -- MORE -- forward: , or j backward: b or k quit: q CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan]  -- MORE -- forward: , or j backward: b or k quit: q CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 66s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ  -- MORE -- forward: , or j backward: b or k quit: q ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0  -- MORE -- forward: , or j backward: b or k quit: q  [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20  -- MORE -- forward: , or j backward: b or k quit: q  [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 66s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan]  -- MORE -- forward: , or j backward: b or k quit: q Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio  -- MORE -- forward: , or j backward: b or k quit: q n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11:  -- MORE -- forward: , or j backward: b or k quit: q Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1c/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040  -- MORE -- forward: , or j backward: b or k quit: q R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780  -- MORE -- forward: , or j backward: b or k quit: q R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8  -- MORE -- forward: , or j backward: b or k quit: q RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0  -- MORE -- forward: , or j backward: b or k quit: q  [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0  -- MORE -- forward: , or j backward: b or k quit: q  [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400  -- MORE -- forward: , or j backward: b or k quit: q Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 66s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010  -- MORE -- forward: , or j backward: b or k quit: q RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0  -- MORE -- forward: , or j backward: b or k quit: q R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b0/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20  -- MORE -- forward: , or j backward: b or k quit: q  [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc]  -- MORE -- forward: , or j backward: b or k quit: q  [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace:  -- MORE -- forward: , or j backward: b or k quit: q  [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U  -- MORE -- forward: , or j backward: b or k quit: q ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10  -- MORE -- forward: , or j backward: b or k quit: q  [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100  -- MORE -- forward: , or j backward: b or k quit: q  [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 66s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50  -- MORE -- forward: , or j backward: b or k quit: q  [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729]  -- MORE -- forward: , or j backward: b or k quit: q Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703]  -- MORE -- forward: , or j backward: b or k quit: q Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0  -- MORE -- forward: , or j backward: b or k quit: q  [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1c/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre]  -- MORE -- forward: , or j backward: b or k quit: q  [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b  -- MORE -- forward: , or j backward: b or k quit: q CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0:  -- MORE -- forward: , or j backward: b or k quit: q Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90  -- MORE -- forward: , or j backward: b or k quit: q  [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ  -- MORE -- forward: , or j backward: b or k quit: q ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan]  -- MORE -- forward: , or j backward: b or k quit: q Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan]  -- MORE -- forward: , or j backward: b or k quit: q Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ  -- MORE -- forward: , or j backward: b or k quit: q ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0  -- MORE -- forward: , or j backward: b or k quit: q  [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100  -- MORE -- forward: , or j backward: b or k quit: q  [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296  -- MORE -- forward: , or j backward: b or k quit: q R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio  -- MORE -- forward: , or j backward: b or k quit: q n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8  -- MORE -- forward: , or j backward: b or k quit: q RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400  -- MORE -- forward: , or j backward: b or k quit: q Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400  -- MORE -- forward: , or j backward: b or k quit: q Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400  -- MORE -- forward: , or j backward: b or k quit: q FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio  -- MORE -- forward: , or j backward: b or k quit: q n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80  -- MORE -- forward: , or j backward: b or k quit: q  [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0  -- MORE -- forward: , or j backward: b or k quit: q  [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b0/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0  -- MORE -- forward: , or j backward: b or k quit: q DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace:  -- MORE -- forward: , or j backward: b or k quit: q  [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480  -- MORE -- forward: , or j backward: b or k quit: q  [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150  -- MORE -- forward: , or j backward: b or k quit: q  [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc]  -- MORE -- forward: , or j backward: b or k quit: q  [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010  -- MORE -- forward: , or j backward: b or k quit: q RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ  -- MORE -- forward: , or j backward: b or k quit: q ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160  -- MORE -- forward: , or j backward: b or k quit: q  [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200  -- MORE -- forward: , or j backward: b or k quit: q  [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1c/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12:  -- MORE -- forward: , or j backward: b or k quit: q Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10:  -- MORE -- forward: , or j backward: b or k quit: q Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U)  -- MORE -- forward: , or j backward: b or k quit: q lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220  -- MORE -- forward: , or j backward: b or k quit: q  [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30  -- MORE -- forward: , or j backward: b or k quit: q  [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8  -- MORE -- forward: , or j backward: b or k quit: q RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach  -- MORE -- forward: , or j backward: b or k quit: q e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ  -- MORE -- forward: , or j backward: b or k quit: q ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78  -- MORE -- forward: , or j backward: b or k quit: q R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001  -- MORE -- forward: , or j backward: b or k quit: q R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30  -- MORE -- forward: , or j backward: b or k quit: q RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde  -- MORE -- forward: , or j backward: b or k quit: q  [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 66s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220  -- MORE -- forward: , or j backward: b or k quit: q  [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace:  -- MORE -- forward: , or j backward: b or k quit: q  [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286  -- MORE -- forward: , or j backward: b or k quit: q RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1c/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b  -- MORE -- forward: , or j backward: b or k quit: q FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0  -- MORE -- forward: , or j backward: b or k quit: q  [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 66s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc]  -- MORE -- forward: , or j backward: b or k quit: q  [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc]  -- MORE -- forward: , or j backward: b or k quit: q  [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio  -- MORE -- forward: , or j backward: b or k quit: q n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370  -- MORE -- forward: , or j backward: b or k quit: q  [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b0/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180  -- MORE -- forward: , or j backward: b or k quit: q  [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20  -- MORE -- forward: , or j backward: b or k quit: q  [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 66s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 66s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ  -- MORE -- forward: , or j backward: b or k quit: q ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ  -- MORE -- forward: , or j backward: b or k quit: q ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0  -- MORE -- forward: , or j backward: b or k quit: q  [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre]  -- MORE -- forward: , or j backward: b or k quit: q  [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0  -- MORE -- forward: , or j backward: b or k quit: q DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ  -- MORE -- forward: , or j backward: b or k quit: q ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde  -- MORE -- forward: , or j backward: b or k quit: q BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 66s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U)  -- MORE -- forward: , or j backward: b or k quit: q lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8  -- MORE -- forward: , or j backward: b or k quit: q RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8  -- MORE -- forward: , or j backward: b or k quit: q RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U)  -- MORE -- forward: , or j backward: b or k quit: q lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370  -- MORE -- forward: , or j backward: b or k quit: q  [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150  -- MORE -- forward: , or j backward: b or k quit: q  [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020  -- MORE -- forward: , or j backward: b or k quit: q FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach  -- MORE -- forward: , or j backward: b or k quit: q e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30  -- MORE -- forward: , or j backward: b or k quit: q RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace:  -- MORE -- forward: , or j backward: b or k quit: q  [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace:  -- MORE -- forward: , or j backward: b or k quit: q  [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000  -- MORE -- forward: , or j backward: b or k quit: q CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach  -- MORE -- forward: , or j backward: b or k quit: q e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90  -- MORE -- forward: , or j backward: b or k quit: q  [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170  -- MORE -- forward: , or j backward: b or k quit: q  [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80 [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde LustreError: 11-0: an error occurred while communicating with 172.19.4.35@o2ib. The ost_read operation failed with -107 Lustre: lustre2-OST0022-osc-ffff88101ac9c800: Connection to service lustre2-OST0022 via nid 172.19.4.35@o2ib was lost; in progress operations using this service will wait for r ecovery to complete. LustreError: 167-0: This client was evicted by lustre2-OST0022; in progress operations using this service will fail. Lustre: lustre2-OST0022-osc-ffff88101ac9c800: Connection restored to service lustre2-OST0022 using nid 172.19.4.35@o2ib. BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8  -- MORE -- forward: , or j backward: b or k quit: q R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b  -- MORE -- forward: , or j backward: b or k quit: q CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110  -- MORE -- forward: , or j backward: b or k quit: q  [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x21/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60  -- MORE -- forward: , or j backward: b or k quit: q  [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 67s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass]  -- MORE -- forward: , or j backward: b or k quit: q  [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan]  -- MORE -- forward: , or j backward: b or k quit: q Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130  -- MORE -- forward: , or j backward: b or k quit: q  [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#0 stuck for 67s! [sshd:57858] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 0: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57858, comm: sshd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff8809a179b7d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8809a179b818 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000296 R13: ffff8809a179b798 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f04f76047c0(0000) GS:ffff88004ac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f04f8a03000 CR3: 0000000801225000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? kmem_getpages+0x62/0x170 [] ? fallback_alloc+0x1ba/0x270 [] ? ____cache_alloc_node+0x99/0x160 [] ? __alloc_skb+0x7a/0x180 [] ? kmem_cache_alloc_node_notrace+0x6f/0x130 [] ? __kmalloc_node+0x7b/0x100 [] ? __alloc_skb+0x7a/0x180 [] ? sock_alloc_send_pskb+0x1c4/0x320 [] ? __wake_up_sync_key+0x5e/0x80  -- MORE -- forward: , or j backward: b or k quit: q  [] ? sock_alloc_send_skb+0x15/0x20 [] ? unix_stream_sendmsg+0x2a0/0x420 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? sock_aio_write+0x151/0x160 [] ? do_sync_write+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? audit_syscall_entry+0x272/0x2a0 [] ? security_file_permission+0x16/0x20 [] ? vfs_write+0x184/0x1a0 [] ? sys_write+0x51/0x90 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#6 stuck for 67s! [khugepaged:181] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 6: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 181, comm: khugepaged Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff880817aa3b90 EFLAGS: 00000202 RAX: 0000000000000010 RBX: ffff880817aa3bd0 RCX: 0000000000000010 RDX: 000000000000000f RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88000002dbf8 R13: 0000000000000286 R14: ffff880817aa3b50 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88004acc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031470abd60 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? alloc_pages_vma+0x9a/0x150  -- MORE -- forward: , or j backward: b or k quit: q  [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x809/0x1200 [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1200 [] ? kthread+0x96/0xa0 [] ? child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#11 stuck for 67s! [ll_ping:3232] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 11: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3232, comm: ll_ping Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff881018e45e40 EFLAGS: 00000297 RAX: 00000000000066dc RBX: ffff881018e45e40 RCX: 00000000ffff9175 RDX: 00000000000066db RSI: 0000000000000282 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000001 R10: 000000000000000a R11: 0000000000000000 R12: ffff881018e45df0 R13: 0000000000000000 R14: 0000000000000009 R15: 000000000000002b FS: 00007f4da7f0b700(0000) GS:ffff88085c460000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000008f4945 CR3: 0000000001a25000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_pinger_main+0x20f/0x8b0 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? child_rip+0xa/0x20 [] ? ptlrpc_pinger_main+0x0/0x8b0 [ptlrpc] [] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#12 stuck for 67s! [sh:57729] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U  -- MORE -- forward: , or j backward: b or k quit: q ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 12: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57729, comm: sh Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff88009b87dbf0 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff88009b87dc30 RCX: 0000000000000010 RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: ffff880840021b78 R13: 0000000000000286 R14: ffff88009b87dbb0 R15: 0000000000000001 FS: 00007ffdc1dbf700(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000014c1018 CR3: 00000000690f1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? path_to_nameidata+0x25/0x60 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? mntput_no_expire+0x30/0x110 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1300 [] ? rb_reserve_next_event+0xb4/0x370 [] ? do_fork+0x94/0x480 [] ? audit_syscall_entry+0x272/0x2a0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#10 stuck for 67s! [cmahostd:3703] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U  -- MORE -- forward: , or j backward: b or k quit: q ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 10: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 3703, comm: cmahostd Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff8807fadc39d8 EFLAGS: 00000293 RAX: 00000000000066dd RBX: ffff8807fadc39d8 RCX: 0000000000000028 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000004 R09: ffff88100da0b780 R10: ffff8807fadc3a38 R11: 0000000000000000 R12: 0000000000000001 R13: ffff880840021b48 R14: ffff8807f2981918 R15: ffff880700000041 FS: 00007f22a02e6700(0000) GS:ffff88085c440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22a0a07000 CR3: 00000007fcf94000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? __mem_cgroup_try_charge+0x78/0x420 [] ? lustre_msg_set_opc+0xac/0x120 [ptlrpc] [] ? ptlrpc_at_set_req_timeout+0x56/0xf0 [ptlrpc] [] ? ptlrpc_prep_req_pool+0x306/0x780 [ptlrpc] [] ? mdc_statfs+0x101/0x4a0 [mdc] [] ? handle_pte_fault+0x487/0xb50 [] ? proc_delete_inode+0x0/0x60 [] ? proc_delete_inode+0x0/0x60 [] ? ll_statfs_internal+0x10c/0x990 [lustre] [] ? put_dec+0x10c/0x110 [] ? dput+0x9a/0x150 [] ? dput+0x9a/0x150 [] ? __follow_mount+0x69/0xb0 [] ? ll_statfs+0x87/0x140 [lustre] [] ? mntput_no_expire+0x30/0x110 [] ? vfs_statfs+0x7c/0xa0 [] ? vfs_statfs_native+0x20/0xb0 [] ? sys_statfs+0x73/0xb0 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0  -- MORE -- forward: , or j backward: b or k quit: q  [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#2 stuck for 66s! [bash:57737] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 2: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 57737, comm: bash Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff880ef3a875c8 EFLAGS: 00000293 RAX: 00000000000066de RBX: ffff880ef3a875c8 RCX: 0000000000000038 RDX: 00000000000066db RSI: 0000000000000000 RDI: ffff8807ff5b8250 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000065 R13: ffff8807e80cec00 R14: 0000000000000000 R15: ffff8805642bf400 FS: 00007f619fe94700(0000) GS:ffff88004ac40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006d6630 CR3: 000000077dc19000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? ptlrpc_queue_wait+0xd8/0x1770 [ptlrpc] [] ? ldlm_resource_putref_internal+0xb3/0x4c0 [ptlrpc] [] ? class_handle_hash+0xad/0x2a0 [obdclass] [] ? ldlm_lock_remove_from_lru+0x44/0x110 [ptlrpc] [] ? ldlm_cli_enqueue+0x2f8/0x7a0 [ptlrpc] [] ? mdc_enqueue+0x3b5/0xa20 [mdc] [] ? ldlm_lock_add_to_lru+0x49/0x110 [ptlrpc] [] ? mdc_intent_lock+0x102/0x440 [mdc] [] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] [] ? ldlm_lock_decref+0x41/0xb0 [ptlrpc] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] [] ? ll_lookup_it+0x315/0x870 [lustre] [] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] [] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre]  -- MORE -- forward: , or j backward: b or k quit: q  [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [] ? ll_lookup_nd+0x1bf/0x470 [lustre] [] ? d_alloc+0x13e/0x1b0 [] ? do_lookup+0x18b/0x220 [] ? __link_path_walk+0x1d3/0x820 [] ? do_follow_link+0x198/0x440 [] ? __link_path_walk+0x20b/0x820 [] ? path_walk+0x6a/0xe0 [] ? do_path_lookup+0x5b/0xa0 [] ? user_path_at+0x57/0xa0 [] ? rb_reserve_next_event+0xb4/0x370 [] ? ring_buffer_lock_reserve+0xa2/0x160 [] ? vfs_fstatat+0x3c/0x80 [] ? vfs_stat+0x1b/0x20 [] ? sys_newstat+0x24/0x50 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0x7a/0xde [] ? tracesys+0xd9/0xde BUG: soft lockup - CPU#9 stuck for 67s! [pg_basebackup:31037] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] CPU 9: Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 31037, comm: pg_basebackup Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] smp_call_function_many+0x1b2/0x210 RSP: 0018:ffff8808685ad5d8 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffff8808685ad618 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: ffffffff81b9e088 R09: 0000000000000040 R10: 0000000000000001 R11: ffff880819e84268 R12: 0000000000000296 R13: ffff8808685ad598 R14: 0000000000000001 R15: 0000000000000020 FS: 00007ff880844700(0000) GS:ffff88085c420000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff874400000 CR3: 00000008680a1000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400  -- MORE -- forward: , or j backward: b or k quit: q Call Trace: [] ? smp_call_function_many+0x19a/0x210 [] ? drain_local_pages+0x0/0x20 [] ? smp_call_function+0x22/0x30 [] ? on_each_cpu+0x24/0x50 [] ? drain_all_pages+0x1c/0x20 [] ? __alloc_pages_nodemask+0x50c/0x8b0 [] ? nf_nat_get_offset+0x62/0xa0 [nf_nat] [] ? ipoib_cm_send+0x100/0x3b0 [ib_ipoib] [] ? swiotlb_map_page+0x0/0x100 [] ? alloc_pages_vma+0x9a/0x150 [] ? do_huge_pmd_anonymous_page+0x13e/0x350 [] ? handle_mm_fault+0x24a/0x2a0 [] ? __do_page_fault+0x139/0x480 [] ? trace_nowake_buffer_unlock_commit+0x43/0x60 [] ? ftrace_raw_event_timer_start+0xca/0xd0 [] ? mod_timer+0x144/0x220 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_string+0x2d/0x40 [] ? memcpy_toiovec+0x55/0x80 [] ? skb_copy_datagram_iovec+0x60/0x2c0 [] ? tcp_recvmsg+0xc95/0xe80 [] ? inet_recvmsg+0x5c/0x90 [] ? sock_recvmsg+0x133/0x160 [] ? autoremove_wake_function+0x0/0x40 [] ? common_interrupt+0xe/0x13 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] ? rb_reserve_next_event+0xb4/0x370 [] ? sys_recvfrom+0xee/0x180 [] ? ftrace_raw_event_sys_enter+0xd9/0x130 [] ? syscall_trace_enter+0x1d8/0x1e0 [] ? tracesys+0xd9/0xde SysRq : Emergency Sync SysRq : Emergency Sync SysRq : Emergency Sync SysRq : Emergency Remount R/O SysRq : Trigger a crash BUG: unable to handle kernel NULL pointer dereference at (null) IP: [] sysrq_handle_crash+0x16/0x20 PGD 959de3067 PUD 959d22067 PMD 0 Oops: 0002 [#1] SMP last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map CPU 3  -- MORE -- forward: , or j backward: b or k quit: q Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Modules linked in: xfs exportfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJ ECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc mptctl mptbase mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) dm_round_robin autofs4 ipmi_devintf ipmi_si ipmi_msghandler ib_srp(U) scsi_transport_srp scsi_tgt sunrpc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U ) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_regio n_hash dm_log vhost_net macvtap macvlan tun kvm_intel kvm uinput power_meter hwmon hpilo hpwdt igb(U) dca sg microcode serio_raw iTCO_wdt iTCO_vendor_support shpchp ext4 mbcach e jbd2 sd_mod crc_t10dif hpsa(U) mpt2sas(U) scsi_transport_sas raid_class ata_generic pata_acpi ata_piix dm_multipath dm_mod [last unloaded: scsi_wait_scan] Pid: 0, comm: swapper Not tainted 2.6.32-131.0.15.el6.x86_64 #1 ProLiant SL230s Gen8 RIP: 0010:[] [] sysrq_handle_crash+0x16/0x20 RSP: 0018:ffff88004ac63db8 EFLAGS: 00010096 RAX: 0000000000000010 RBX: 0000000000000063 RCX: 0000000000009d52 RDX: 0000000000000000 RSI: ffff8808175ad000 RDI: 0000000000000063 RBP: ffff88004ac63db8 R08: 0000000000000073 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000000 R12: ffff8808175ad000 R13: ffffffff81a99e80 R14: 0000000000000082 R15: 0000000000000007 FS: 0000000000000000(0000) GS:ffff88004ac60000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000000000000000 CR3: 000000095c622000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper (pid: 0, threadinfo ffff88101c27a000, task ffff88081cb72080) Stack: ffff88004ac63e08 ffffffff8131ae82 0000000400000000 0000000000000001 <0> 0000000000000003 ffffffff81f2cdb8 00000000000000f2 0000000000000061 <0> ffff8808175ad000 0000000000000296 ffff88004ac63e18 ffffffff8131af6b Call Trace: [] __handle_sysrq+0x132/0x1a0 [] handle_sysrq+0x2b/0x30 [] serial8250_handle_port+0x218/0x320 [] serial8250_interrupt+0x8b/0x120 [] handle_IRQ_event+0x60/0x170 [] ? sched_clock+0x9/0x10 [] handle_edge_irq+0xc6/0x160 [] ? tick_check_idle+0xdb/0xe0 [] handle_irq+0x49/0xa0 [] do_IRQ+0x6c/0xf0 [] ret_from_intr+0x0/0x11  -- MORE -- forward: , or j backward: b or k quit: q  [] ? mwait_idle+0x77/0xd0 [] ? mwait_idle+0xc0/0xd0 [] cpu_idle+0xb6/0x110 [] start_secondary+0x202/0x245 Code: d0 88 81 63 a2 f0 81 c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 c7 05 2d c5 71 00 01 00 00 00 0f ae f8 04 25 00 00 00 00 01 c9 c3 55 48 89 e5 0f 1f 44 00 00 8d 47 RIP [] sysrq_handle_crash+0x16/0x20 RSP CR2: 0000000000000000 [?1l>crash> crash> crash> exit ]0;ddnuser01@wk5:~/fromDDBJ[ddnuser01@wk5 fromDDBJ]$ ]0;ddnuser01@wk5:~/fromDDBJ[ddnuser01@wk5 fromDDBJ]$ exit exit スクリプトは 2013年03月19日 20時08分34秒 に終了しました