[LU-4535] Kernel panic - not syncing: softlockup: hung tasks

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.1.6
• Type: Bug
• Priority: Major
• Reporter: Roger Spellman (Inactive)
• Assignee: John Fuchs-Chesney (Inactive)
• Resolution: Won't Fix

Environment

We are running Lustre 2.1.5.
Each MDS has 32 cores of Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz, with 128 GB of RAM.

• Severity: 2

Description

We have a system that panics every time the MDT is under very heavy load. Below is a soft lockup, immediately followed by a panic.

We were also running ‘top’, to see what is going on. The 32 kiblnd_sd tasks are very busy, as are the 72 mdt_rdpg tasks. I have put the top output below.

We are running Lustre 2.1.5. Each MDS has 32 cores of Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz, with 128 GB of RAM.

Since this panic occurs quite regularly, I’d really like around or a fix. Could decreasing the number of threads help? Which threads?

Thanks.

ibwarn: [4816] _do_madrpc: recv failed: Connecti
BUG: soft lockup - CPU#0 stuck for 61s! [mdt_95:18762]
Modules linked in: cmm(U) osd_ldiskfs(U) mdt(U) mdd(U) mds(U) fsfilt_ldiskfs(U) exportfs mgs(U) mgc(U) ldiskfs(U) lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) ipmi_si usbhid uhci_hcd ohci_hcd ehci_hcd dell_rbu 8021q garp stp llc ipmi_devintf ipmi_msghandler ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa mlx4_ib ib_mad ib_core rtc_cmos rtc_core rtc_lib sg thermal processor thermal_sys dcdbas scsi_dh_rdac mpt2sas scsi_transport_sas raid_class mlx4_en mlx4_core megaraid_sas qla4xxx iscsi_boot_sysfs libiscsi [last unloaded: ipmi_si]
CPU 0
Modules linked in: cmm(U) osd_ldiskfs(U) mdt(U) mdd(U) mds(U) fsfilt_ldiskfs(U) exportfs mgs(U) mgc(U) ldiskfs(U) lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) ipmi_si usbhid uhci_hcd ohci_hcd ehci_hcd dell_rbu 8021q garp stp llc ipmi_devintf ipmi_msghandler ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa mlx4_ib ib_mad ib_core rtc_cmos rtc_core rtc_lib sg thermal processor thermal_sys dcdbas scsi_dh_rdac mpt2sas scsi_transport_sas raid_class mlx4_en mlx4_core megaraid_sas qla4xxx iscsi_boot_sysfs libiscsi [last unloaded: ipmi_si]

Pid: 18762, comm: mdt_95 Not tainted 2.6.32-279.19.1.el6_lustre.2.1.5_1.0.7 #1 Dell Inc. PowerEdge R620/0KCKR5
RIP: 0010:[<ffffffff814180fa>] [<ffffffff814180fa>] skb_release_data+0xd4/0xd9
RSP: 0018:ffff880061603950 EFLAGS: 00000286
RAX: 000000000000002b RBX: ffff880061603960 RCX: 0000000000000001
RDX: 000000000000002a RSI: ffff8800616039c0 RDI: ffff880ee57d0e00
RBP: ffffffff81003573 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000080 R11: 0000000000000080 R12: ffff8800616038d0
R13: 0000000000000000 R14: ffff880fee47b7c0 R15: ffffffff814d67b7
FS: 00007fb57d425700(0000) GS:ffff880061600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000332fadf200 CR3: 0000000001b13000 CR4: 00000000000406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mdt_95 (pid: 18762, threadinfo ffff881e5da8c000, task ffff881e488ca870)
Stack:
ffff880fee47b7c0 ffff881015052020 ffff880061603980 ffffffff814187dd
<d> ffff880fee47b7c0 ffff880fee47b7c0 ffff8800616039a0 ffffffff81417d6d
<d> ffff8800616039d0 ffff880fee47b7c0 ffff8800616039c0 ffffffff81417e31
Call Trace:
<IRQ>
[<ffffffff814187dd>] ? skb_release_all+0x1e/0x22
[<ffffffff81417d6d>] ? __kfree_skb+0x16/0x7d
[<ffffffff81417e31>] ? consume_skb+0x5d/0x62
[<ffffffff81422ab2>] ? dev_kfree_skb_any+0x33/0x35
[<ffffffffa018b974>] ? poll_tx+0xc8/0x183 [ib_ipoib]
[<ffffffffa018d378>] ? ipoib_send+0x72b/0x73e [ib_ipoib]
[<ffffffffa018b155>] ? ipoib_start_xmit+0x333/0x39e [ib_ipoib]
[<ffffffff81420ae6>] ? dev_hard_start_xmit+0x10e/0x245
[<ffffffff8143658e>] ? sch_direct_xmit+0x66/0x15c
[<ffffffff81423d4e>] ? dev_queue_xmit+0x39d/0x50b
[<ffffffff8146a456>] ? arp_xmit+0x53/0x57
[<ffffffff8146ad42>] ? arp_send+0x39/0x3b
[<ffffffff8146b050>] ? arp_process+0x30c/0x5fd
[<ffffffff8146b445>] ? arp_rcv+0x104/0x118
[<ffffffff8141fdda>] ? __netif_receive_skb+0x4d0/0x51d
[<ffffffff8141fe70>] ? netif_receive_skb+0x49/0x50
[<ffffffff81420347>] ? __napi_gro_receive+0x101/0x10d
[<ffffffff8141ff28>] ? napi_skb_finish+0x29/0x40
[<ffffffff814203c9>] ? napi_gro_receive+0x2f/0x3d
[<ffffffffa018c1b7>] ? ipoib_ib_handle_rx_wc+0x216/0x249 [ib_ipoib]
[<ffffffffa018d4ba>] ? ipoib_poll+0x99/0x123 [ib_ipoib]
[<ffffffff81422b77>] ? net_rx_action+0xc3/0x252
[<ffffffff81049426>] ? __do_softirq+0xda/0x199
[<ffffffff81092bd9>] ? handle_IRQ_event+0x60/0x114
[<ffffffff81003b9c>] ? call_softirq+0x1c/0x28
[<ffffffff810051a7>] ? do_softirq+0x38/0x71
[<ffffffff8104930d>] ? irq_exit+0x3b/0x7a
[<ffffffff814d6718>] ? do_IRQ+0xa8/0xbf
[<ffffffff81003413>] ? ret_from_intr+0x0/0xa
<EOI>
[<ffffffffa02f171c>] ? class_handle2object+0x6c/0x170 [obdclass]
[<ffffffffa042e4e6>] ? __ldlm_handle2lock+0x46/0x340 [ptlrpc]
[<ffffffffa044dcd6>] ? ldlm_request_cancel+0x176/0x3b0 [ptlrpc]
[<ffffffffa044e1c6>] ? ldlm_handle_enqueue0+0x66/0xf40 [ptlrpc]
[<ffffffffa04762b0>] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc]
[<ffffffffa0af7e06>] ? mdt_enqueue+0x46/0x130 [mdt]
[<ffffffffa0aecd49>] ? mdt_handle_common+0xb79/0x1730 [mdt]
[<ffffffffa0aed9d5>] ? mdt_regular_handle+0x15/0x20 [mdt]
[<ffffffffa0480471>] ? ptlrpc_server_handle_request+0x4d1/0xfa0 [ptlrpc]
[<ffffffff81050d6c>] ? mod_timer+0x188/0x19a
[<ffffffffa01ef1de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
[<ffffffffa01fc21c>] ? lc_watchdog_touch+0x6c/0x180 [libcfs]
[<ffffffffa0483934>] ? ptlrpc_wait_event+0xa4/0x2d0 [ptlrpc]
[<ffffffff81037fd7>] ? default_wake_function+0x0/0x14
[<ffffffffa04842be>] ? ptlrpc_main+0x75e/0x16f0 [ptlrpc]
[<ffffffff81003a9a>] ? child_rip+0xa/0x20
[<ffffffffa0483b60>] ? ptlrpc_main+0x0/0x16f0 [ptlrpc]
[<ffffffff81003a90>] ? child_rip+0x0/0x20
Code: ff d0 8b 83 b8 00 00 00 48 03 83 c0 00 00 00 48 83 78 18 00 74 09 48 8d 78 18 e8 06 ff ff ff 48 8b bb c0 00 00 00 e8 e4 37 ce ff <5b> 41 5c c9 c3 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec
Call Trace:
<IRQ> [<ffffffff814187dd>] ? skb_release_all+0x1e/0x22
[<ffffffff81417d6d>] ? __kfree_skb+0x16/0x7d
[<ffffffff81417e31>] ? consume_skb+0x5d/0x62
[<ffffffff81422ab2>] ? dev_kfree_skb_any+0x33/0x35
[<ffffffffa018b974>] ? poll_tx+0xc8/0x183 [ib_ipoib]
[<ffffffffa018d378>] ? ipoib_send+0x72b/0x73e [ib_ipoib]
[<ffffffffa018b155>] ? ipoib_start_xmit+0x333/0x39e [ib_ipoib]
[<ffffffff81420ae6>] ? dev_hard_start_xmit+0x10e/0x245
[<ffffffff8143658e>] ? sch_direct_xmit+0x66/0x15c
[<ffffffff81423d4e>] ? dev_queue_xmit+0x39d/0x50b
[<ffffffff8146a456>] ? arp_xmit+0x53/0x57
[<ffffffff8146ad42>] ? arp_send+0x39/0x3b
[<ffffffff8146b050>] ? arp_process+0x30c/0x5fd
[<ffffffff8146b445>] ? arp_rcv+0x104/0x118
[<ffffffff8141fdda>] ? __netif_receive_skb+0x4d0/0x51d
[<ffffffff8141fe70>] ? netif_receive_skb+0x49/0x50
[<ffffffff81420347>] ? __napi_gro_receive+0x101/0x10d
[<ffffffff8141ff28>] ? napi_skb_finish+0x29/0x40
[<ffffffff814203c9>] ? napi_gro_receive+0x2f/0x3d
[<ffffffffa018c1b7>] ? ipoib_ib_handle_rx_wc+0x216/0x249 [ib_ipoib]
[<ffffffffa018d4ba>] ? ipoib_poll+0x99/0x123 [ib_ipoib]
[<ffffffff81422b77>] ? net_rx_action+0xc3/0x252
[<ffffffff81049426>] ? __do_softirq+0xda/0x199
[<ffffffff81092bd9>] ? handle_IRQ_event+0x60/0x114
[<ffffffff81003b9c>] ? call_softirq+0x1c/0x28
[<ffffffff810051a7>] ? do_softirq+0x38/0x71
[<ffffffff8104930d>] ? irq_exit+0x3b/0x7a
[<ffffffff814d6718>] ? do_IRQ+0xa8/0xbf
[<ffffffff81003413>] ? ret_from_intr+0x0/0xa
<EOI> [<ffffffffa02f171c>] ? class_handle2object+0x6c/0x170 [obdclass]
[<ffffffffa042e4e6>] ? __ldlm_handle2lock+0x46/0x340 [ptlrpc]
[<ffffffffa044dcd6>] ? ldlm_request_cancel+0x176/0x3b0 [ptlrpc]
[<ffffffffa044e1c6>] ? ldlm_handle_enqueue0+0x66/0xf40 [ptlrpc]
[<ffffffffa04762b0>] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc]
[<ffffffffa0af7e06>] ? mdt_enqueue+0x46/0x130 [mdt]
[<ffffffffa0aecd49>] ? mdt_handle_common+0xb79/0x1730 [mdt]
[<ffffffffa0aed9d5>] ? mdt_regular_handle+0x15/0x20 [mdt]
[<ffffffffa0480471>] ? ptlrpc_server_handle_request+0x4d1/0xfa0 [ptlrpc]
[<ffffffff81050d6c>] ? mod_timer+0x188/0x19a
[<ffffffffa01ef1de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
[<ffffffffa01fc21c>] ? lc_watchdog_touch+0x6c/0x180 [libcfs]
[<ffffffffa0483934>] ? ptlrpc_wait_event+0xa4/0x2d0 [ptlrpc]
[<ffffffff81037fd7>] ? default_wake_function+0x0/0x14
[<ffffffffa04842be>] ? ptlrpc_main+0x75e/0x16f0 [ptlrpc]
[<ffffffff81003a9a>] ? child_rip+0xa/0x20
[<ffffffffa0483b60>] ? ptlrpc_main+0x0/0x16f0 [ptlrpc]
[<ffffffff81003a90>] ? child_rip+0x0/0x20
Kernel panic - not syncing: softlockup: hung tasks
Pid: 18762, comm: mdt_95 Not tainted 2.6.32-279.19.1.el6_lustre.2.1.5_1.0.7 #1
Call Trace:
<IRQ> [<ffffffff810424d4>] ? panic+0xd3/0x1aa
[<ffffffff81003a90>] ? child_rip+0x0/0x20
[<ffffffff810926c9>] ? watchdog_timer_fn+0x0/0x17f
[<ffffffff81006305>] ? show_trace_log_lvl+0x45/0x4e
[<ffffffff81006323>] ? show_trace+0x15/0x17
[<ffffffff81001465>] ? show_regs+0x27/0x2b
[<ffffffff81092817>] ? watchdog_timer_fn+0x14e/0x17f
[<ffffffff81064b2a>] ? __run_hrtimer+0xad/0x10b
[<ffffffff81064d7a>] ? hrtimer_interrupt+0xb3/0x199
[<ffffffff814d67b7>] ? smp_apic_timer_interrupt+0x88/0x9b
[<ffffffff81003573>] ? apic_timer_interrupt+0x13/0x20
[<ffffffff814180fa>] ? skb_release_data+0xd4/0xd9
[<ffffffff814187dd>] ? skb_release_all+0x1e/0x22
[<ffffffff81417d6d>] ? __kfree_skb+0x16/0x7d
[<ffffffff81417e31>] ? consume_skb+0x5d/0x62
[<ffffffff81422ab2>] ? dev_kfree_skb_any+0x33/0x35
[<ffffffffa018b974>] ? poll_tx+0xc8/0x183 [ib_ipoib]
[<ffffffffa018d378>] ? ipoib_send+0x72b/0x73e [ib_ipoib]
[<ffffffffa018b155>] ? ipoib_start_xmit+0x333/0x39e [ib_ipoib]
[<ffffffff81420ae6>] ? dev_hard_start_xmit+0x10e/0x245
[<ffffffff8143658e>] ? sch_direct_xmit+0x66/0x15c
[<ffffffff81423d4e>] ? dev_queue_xmit+0x39d/0x50b
[<ffffffff8146a456>] ? arp_xmit+0x53/0x57
[<ffffffff8146ad42>] ? arp_send+0x39/0x3b
[<ffffffff8146b050>] ? arp_process+0x30c/0x5fd
[<ffffffff8146b445>] ? arp_rcv+0x104/0x118
[<ffffffff8141fdda>] ? __netif_receive_skb+0x4d0/0x51d
[<ffffffff8141fe70>] ? netif_receive_skb+0x49/0x50
[<ffffffff81420347>] ? __napi_gro_receive+0x101/0x10d
[<ffffffff8141ff28>] ? napi_skb_finish+0x29/0x40
[<ffffffff814203c9>] ? napi_gro_receive+0x2f/0x3d
[<ffffffffa018c1b7>] ? ipoib_ib_handle_rx_wc+0x216/0x249 [ib_ipoib]
[<ffffffffa018d4ba>] ? ipoib_poll+0x99/0x123 [ib_ipoib]
[<ffffffff81422b77>] ? net_rx_action+0xc3/0x252
[<ffffffff81049426>] ? __do_softirq+0xda/0x199
[<ffffffff81092bd9>] ? handle_IRQ_event+0x60/0x114
[<ffffffff81003b9c>] ? call_softirq+0x1c/0x28
[<ffffffff810051a7>] ? do_softirq+0x38/0x71
[<ffffffff8104930d>] ? irq_exit+0x3b/0x7a
[<ffffffff814d6718>] ? do_IRQ+0xa8/0xbf
[<ffffffff81003413>] ? ret_from_intr+0x0/0xa
<EOI> [<ffffffffa02f171c>] ? class_handle2object+0x6c/0x170 [obdclass]
[<ffffffffa042e4e6>] ? __ldlm_handle2lock+0x46/0x340 [ptlrpc]
[<ffffffffa044dcd6>] ? ldlm_request_cancel+0x176/0x3b0 [ptlrpc]
[<ffffffffa044e1c6>] ? ldlm_handle_enqueue0+0x66/0xf40 [ptlrpc]
[<ffffffffa04762b0>] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc]
[<ffffffffa0af7e06>] ? mdt_enqueue+0x46/0x130 [mdt]
[<ffffffffa0aecd49>] ? mdt_handle_common+0xb79/0x1730 [mdt]
[<ffffffffa0aed9d5>] ? mdt_regular_handle+0x15/0x20 [mdt]
[<ffffffffa0480471>] ? ptlrpc_server_handle_request+0x4d1/0xfa0 [ptlrpc]
[<ffffffff81050d6c>] ? mod_timer+0x188/0x19a
[<ffffffffa01ef1de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
[<ffffffffa01fc21c>] ? lc_watchdog_touch+0x6c/0x180 [libcfs]
[<ffffffffa0483934>] ? ptlrpc_wait_event+0xa4/0x2d0 [ptlrpc]
[<ffffffff81037fd7>] ? default_wake_function+0x0/0x14
[<ffffffffa04842be>] ? ptlrpc_main+0x75e/0x16f0 [ptlrpc]
[<ffffffff81003a9a>] ? child_rip+0xa/0x20
[<ffffffffa0483b60>] ? ptlrpc_main+0x0/0x16f0 [ptlrpc]
[<ffffffff81003a90>] ? child_rip+0x0/0x20

top - 10:18:55 up 1 day, 8:52, 2 users, load average: 31.92, 40.97, 55.52
Tasks: 1054 total, 26 running, 1024 sleeping, 0 stopped, 4 zombie
Cpu(s): 0.2%us, 63.8%sy, 0.0%ni, 31.8%id, 4.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 132248176k total, 64036764k used, 68211412k free, 19978956k buffers
Swap: 2076664k total, 0k used, 2076664k free, 1663576k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18730 root 20 0 0 0 0 S 26.9 0.0 16:41.97 kiblnd_sd_22
18723 root 20 0 0 0 0 R 26.8 0.0 16:46.12 kiblnd_sd_15
18734 root 20 0 0 0 0 R 26.8 0.0 16:44.02 kiblnd_sd_26
18718 root 20 0 0 0 0 R 26.8 0.0 16:43.62 kiblnd_sd_10
18720 root 20 0 0 0 0 R 26.7 0.0 16:46.43 kiblnd_sd_12
18737 root 20 0 0 0 0 R 26.6 0.0 16:43.34 kiblnd_sd_29
18726 root 20 0 0 0 0 S 26.6 0.0 16:43.09 kiblnd_sd_18
18712 root 20 0 0 0 0 S 26.5 0.0 16:40.22 kiblnd_sd_04
18732 root 20 0 0 0 0 R 26.5 0.0 16:40.51 kiblnd_sd_24
18735 root 20 0 0 0 0 S 26.5 0.0 16:42.95 kiblnd_sd_27
18713 root 20 0 0 0 0 S 26.5 0.0 16:41.83 kiblnd_sd_05
18736 root 20 0 0 0 0 S 26.5 0.0 16:45.68 kiblnd_sd_28
18719 root 20 0 0 0 0 R 26.5 0.0 16:44.39 kiblnd_sd_11
18733 root 20 0 0 0 0 S 26.5 0.0 16:43.84 kiblnd_sd_25
18738 root 20 0 0 0 0 S 26.5 0.0 16:45.56 kiblnd_sd_30
18711 root 20 0 0 0 0 S 26.4 0.0 16:40.67 kiblnd_sd_03
18716 root 20 0 0 0 0 R 26.4 0.0 16:41.09 kiblnd_sd_08
18717 root 20 0 0 0 0 R 26.4 0.0 16:41.60 kiblnd_sd_09
18708 root 20 0 0 0 0 R 26.4 0.0 16:42.03 kiblnd_sd_00
18724 root 20 0 0 0 0 S 26.4 0.0 16:30.90 kiblnd_sd_16
18731 root 20 0 0 0 0 R 26.4 0.0 16:42.05 kiblnd_sd_23
18709 root 20 0 0 0 0 S 26.3 0.0 16:41.32 kiblnd_sd_01
18714 root 20 0 0 0 0 R 26.3 0.0 16:40.81 kiblnd_sd_06
18725 root 20 0 0 0 0 S 26.2 0.0 16:43.62 kiblnd_sd_17
18722 root 20 0 0 0 0 S 26.1 0.0 16:39.90 kiblnd_sd_14
18727 root 20 0 0 0 0 R 26.0 0.0 16:37.74 kiblnd_sd_19
18729 root 20 0 0 0 0 S 26.0 0.0 16:43.51 kiblnd_sd_21
18721 root 20 0 0 0 0 S 26.0 0.0 16:41.45 kiblnd_sd_13
18739 root 20 0 0 0 0 R 25.9 0.0 16:45.52 kiblnd_sd_31
18715 root 20 0 0 0 0 R 25.8 0.0 16:44.99 kiblnd_sd_07
18710 root 20 0 0 0 0 S 25.5 0.0 16:38.83 kiblnd_sd_02
7740 root 20 0 0 0 0 S 20.0 0.0 6:59.11 mdt_rdpg_54
24971 root 20 0 0 0 0 S 19.9 0.0 9:01.40 mdt_rdpg_32
27377 root 20 0 0 0 0 S 19.9 0.0 9:01.73 mdt_rdpg_42
23919 root 20 0 0 0 0 R 19.8 0.0 8:54.15 mdt_rdpg_28
25101 root 20 0 0 0 0 S 19.7 0.0 9:08.79 mdt_rdpg_35
23918 root 20 0 0 0 0 S 19.6 0.0 9:10.03 mdt_rdpg_27
37779 root 20 0 0 0 0 S 19.5 0.0 3:13.97 mdt_rdpg_70
8613 root 20 0 0 0 0 R 19.4 0.0 6:53.68 mdt_rdpg_66
7829 root 20 0 0 0 0 S 19.2 0.0 6:58.32 mdt_rdpg_60
20457 root 20 0 0 0 0 S 19.2 0.0 9:21.96 mdt_rdpg_19
17628 root 20 0 0 0 0 R 18.9 0.0 9:35.69 mdt_rdpg_17
17629 root 20 0 0 0 0 S 18.9 0.0 9:17.00 mdt_rdpg_18
7824 root 20 0 0 0 0 S 18.9 0.0 6:54.49 mdt_rdpg_55
7825 root 20 0 0 0 0 S 18.7 0.0 6:57.53 mdt_rdpg_56
20015 root 20 0 0 0 0 S 18.6 0.0 9:23.43 mdt_rdpg_07
21202 root 20 0 0 0 0 S 18.5 0.0 9:17.90 mdt_rdpg_22
19746 root 20 0 0 0 0 S 18.3 0.0 9:31.59 mdt_rdpg_04
20158 root 20 0 0 0 0 S 18.1 0.0 9:29.48 mdt_rdpg_09
7994 root 20 0 0 0 0 S 17.7 0.0 6:57.04 mdt_rdpg_63
21548 root 20 0 0 0 0 S 17.5 0.0 8:52.45 mdt_rdpg_23
7737 root 20 0 0 0 0 S 17.4 0.0 6:56.23 mdt_rdpg_52
20458 root 20 0 0 0 0 S 17.1 0.0 9:26.60 mdt_rdpg_20
17420 root 20 0 0 0 0 S 16.6 0.0 9:22.79 mdt_rdpg_13
7826 root 20 0 0 0 0 S 16.4 0.0 6:57.29 mdt_rdpg_57
27373 root 20 0 0 0 0 S 16.2 0.0 8:53.62 mdt_rdpg_40
7831 root 20 0 0 0 0 S 15.6 0.0 6:57.82 mdt_rdpg_61
8292 root 20 0 0 0 0 S 15.5 0.0 6:52.47 mdt_rdpg_65
19282 root 20 0 0 0 0 S 15.2 0.0 9:23.38 mdt_rdpg_01
23920 root 20 0 0 0 0 S 15.2 0.0 9:25.56 mdt_rdpg_29
27374 root 20 0 0 0 0 S 15.0 0.0 9:00.49 mdt_rdpg_41
20014 root 20 0 0 0 0 S 15.0 0.0 9:31.85 mdt_rdpg_06
7738 root 20 0 0 0 0 S 14.7 0.0 6:58.78 mdt_rdpg_53
20159 root 20 0 0 0 0 S 14.6 0.0 9:26.76 mdt_rdpg_10
7995 root 20 0 0 0 0 S 14.6 0.0 6:54.09 mdt_rdpg_64
25100 root 20 0 0 0 0 S 14.4 0.0 9:08.34 mdt_rdpg_34
19718 root 20 0 0 0 0 S 14.4 0.0 9:27.82 mdt_rdpg_02
27380 root 20 0 0 0 0 S 14.2 0.0 8:53.27 mdt_rdpg_45
23921 root 20 0 0 0 0 S 13.8 0.0 9:04.53 mdt_rdpg_30
25099 root 20 0 0 0 0 S 13.6 0.0 9:15.37 mdt_rdpg_33
7827 root 20 0 0 0 0 S 13.6 0.0 6:56.47 mdt_rdpg_58
19729 root 20 0 0 0 0 S 13.5 0.0 11:54.68 mdt_71
27362 root 20 0 0 0 0 D 13.5 0.0 8:28.74 mdt_rdpg_36
7622 root 20 0 0 0 0 S 13.4 0.0 6:56.21 mdt_rdpg_50
9132 root 20 0 0 0 0 S 13.2 0.0 6:48.36 mdt_rdpg_68
19676 root 20 0 0 0 0 S 13.2 0.0 12:05.97 mdt_40
12813 root 20 0 0 0 0 R 12.8 0.0 11:28.13 mdt_87
19602 root 20 0 0 0 0 R 12.8 0.0 11:41.76 mdt_30
19732 root 20 0 0 0 0 R 12.7 0.0 11:54.46 mdt_74
27366 root 20 0 0 0 0 S 12.7 0.0 8:55.50 mdt_rdpg_37
19747 root 20 0 0 0 0 S 12.6 0.0 9:25.21 mdt_rdpg_05
48535 root 20 0 0 0 0 S 12.4 0.0 7:59.08 mdt_rdpg_48
19738 root 20 0 0 0 0 S 12.3 0.0 11:26.04 mdt_80
19710 root 20 0 0 0 0 S 12.2 0.0 11:35.35 mdt_54
23917 root 20 0 0 0 0 S 12.0 0.0 9:21.85 mdt_rdpg_26
37780 root 20 0 0 0 0 R 11.9 0.0 3:13.04 mdt_rdpg_71
21201 root 20 0 0 0 0 S 11.8 0.0 9:21.08 mdt_rdpg_21
27379 root 20 0 0 0 0 S 11.7 0.0 8:49.59 mdt_rdpg_44
8614 root 20 0 0 0 0 S 11.6 0.0 6:53.88 mdt_rdpg_67
7623 root 20 0 0 0 0 S 11.5 0.0 6:55.80 mdt_rdpg_51
19745 root 20 0 0 0 0 S 11.3 0.0 12:02.11 mdt_85
19589 root 20 0 0 0 0 S 11.0 0.0 12:14.64 mdt_17
19594 root 20 0 0 0 0 S 10.4 0.0 11:41.60 mdt_22
19736 root 20 0 0 0 0 R 10.3 0.0 12:24.89 mdt_78
23922 root 20 0 0 0 0 S 9.8 0.0 9:11.15 mdt_rdpg_31
19281 root 20 0 0 0 0 S 9.5 0.0 9:40.34 mdt_rdpg_00
7828 root 20 0 0 0 0 S 9.2 0.0 6:53.57 mdt_rdpg_59
20160 root 20 0 0 0 0 S 9.2 0.0 9:28.40 mdt_rdpg_11
23327 root 20 0 0 0 0 S 8.8 0.0 9:17.97 mdt_rdpg_24
17457 root 20 0 0 0 0 S 8.7 0.0 9:37.37 mdt_rdpg_14
20161 root 20 0 0 0 0 S 8.7 0.0 9:47.07 mdt_rdpg_12
27371 root 20 0 0 0 0 S 8.7 0.0 8:49.93 mdt_rdpg_38
27378 root 20 0 0 0 0 S 8.7 0.0 8:25.97 mdt_rdpg_43
17541 root 20 0 0 0 0 S 8.6 0.0 5:54.36 mdt_rdpg_69
27381 root 20 0 0 0 0 S 8.6 0.0 8:59.66 mdt_rdpg_46
17626 root 20 0 0 0 0 S 8.6 0.0 9:31.03 mdt_rdpg_15
19719 root 20 0 0 0 0 S 8.5 0.0 9:44.04 mdt_rdpg_03
23328 root 20 0 0 0 0 S 8.5 0.0 9:05.57 mdt_rdpg_25
6719 root 20 0 0 0 0 S 8.5 0.0 7:03.25 mdt_rdpg_49
48421 root 20 0 0 0 0 S 8.5 0.0 7:57.04 mdt_rdpg_47
17627 root 20 0 0 0 0 S 8.4 0.0 9:30.86 mdt_rdpg_16
27372 root 20 0 0 0 0 S 8.3 0.0 8:52.46 mdt_rdpg_39
7852 root 20 0 0 0 0 S 8.1 0.0 6:51.36 mdt_rdpg_62
20157 root 20 0 0 0 0 S 8.1 0.0 9:31.37 mdt_rdpg_08
18728 root 20 0 0 0 0 R 8.0 0.0 16:34.60 kiblnd_sd_20
19512 root 20 0 66192 21m 1488 S 5.6 0.0 74:50.82 collectl
19717 root 20 0 0 0 0 D 2.1 0.0 11:48.69 mdt_61
19556 root 20 0 0 0 0 D 1.9 0.0 11:39.91 mdt_03
19715 root 20 0 0 0 0 S 1.7 0.0 12:05.41 mdt_59
13767 root 20 0 0 0 0 D 1.6 0.0 11:06.89 mdt_90
19708 root 20 0 0 0 0 D 1.5 0.0 12:47.88 mdt_52
19574 root 20 0 0 0 0 D 1.5 0.0 11:48.19 mdt_12
19572 root 20 0 0 0 0 D 1.4 0.0 12:16.23 mdt_10
19675 root 20 0 0 0 0 D 1.4 0.0 11:43.10 mdt_39
16746 root 20 0 0 0 0 D 1.4 0.0 10:50.48 mdt_94
19607 root 20 0 0 0 0 D 1.4 0.0 11:50.62 mdt_35
19709 root 20 0 0 0 0 D 1.4 0.0 12:33.96 mdt_53
19723 root 20 0 0 0 0 D 1.2 0.0 11:30.20 mdt_65
19737 root 20 0 0 0 0 D 1.2 0.0 11:34.50 mdt_79
12741 root 20 0 0 0 0 D 1.1 0.0 11:22.51 mdt_86
19716 root 20 0 0 0 0 D 1.1 0.0 12:41.27 mdt_60
17286 root 20 0 0 0 0 S 1.0 0.0 0:22.14 ldlm_cn_24
13315 root 20 0 0 0 0 S 1.0 0.0 11:11.67 mdt_89
19557 root 20 0 0 0 0 D 0.9 0.0 11:17.16 mdt_04
19575 root 20 0 0 0 0 S 0.9 0.0 12:43.76 mdt_13
19599 root 20 0 0 0 0 D 0.9 0.0 12:26.72 mdt_27
19695 root 20 0 0 0 0 S 0.9 0.0 11:46.34 mdt_45
19711 root 20 0 0 0 0 S 0.8 0.0 11:58.66 mdt_55
17281 root 20 0 0 0 0 S 0.8 0.0 0:15.54 ldlm_cn_21
9185 root 20 0 32.5g 70m 9184 S 0.6 0.1 1:27.31 java
19724 root 20 0 0 0 0 S 0.6 0.0 11:15.50 mdt_66
19744 root 20 0 0 0 0 D 0.6 0.0 12:11.37 mdt_84
19569 root 20 0 0 0 0 S 0.6 0.0 11:57.70 mdt_07
19713 root 20 0 0 0 0 D 0.5 0.0 12:03.47 mdt_57
19727 root 20 0 0 0 0 S 0.5 0.0 11:51.88 mdt_69
19972 ganglia 20 0 74000 5628 2284 S 0.4 0.0 1:26.93 gmond
19712 root 20 0 0 0 0 S 0.3 0.0 11:50.30 mdt_56
8152 root 20 0 432m 8028 2872 S 0.2 0.0 1:01.69 dsm_sa_datamgrd
19707 root 20 0 0 0 0 D 0.2 0.0 11:44.10 mdt_51
19725 root 20 0 0 0 0 S 0.2 0.0 11:59.16 mdt_67
7126 root 20 0 0 0 0 D 0.1 0.0 0:44.98 ib_mad1
18547 root 20 0 15828 2012 940 R 0.1 0.0 0:44.02 top
19261 root 20 0 0 0 0 S 0.1 0.0 0:25.45 ll_mgs_01
19262 root 20 0 0 0 0 S 0.1 0.0 0:28.56 ll_mgs_02
19590 root 20 0 0 0 0 D 0.1 0.0 11:36.55 mdt_18
22171 root 20 0 9744 1740 980 S 0.1 0.0 0:09.98 diag.sh
134 root 20 0 0 0 0 S 0.1 0.0 0:07.72 events/3
135 root 20 0 0 0 0 S 0.1 0.0 0:13.53 events/4
136 root 20 0 0 0 0 S 0.1 0.0 0:05.33 events/5
137 root 20 0 0 0 0 S 0.1 0.0 0:13.32 events/6
138 root 20 0 0 0 0 S 0.1 0.0 0:05.59 events/7
141 root 20 0 0 0 0 S 0.1 0.0 0:12.68 events/10
142 root 20 0 0 0 0 S 0.1 0.0 0:05.10 events/11
150 root 20 0 0 0 0 S 0.1 0.0 0:05.08 events/19
153 root 20 0 0 0 0 S 0.1 0.0 0:12.00 events/22
157 root 20 0 0 0 0 S 0.1 0.0 0:12.10 events/26
161 root 20 0 0 0 0 S 0.1 0.0 0:12.74 events/30
7859 root 20 0 267m 9m 5108 S 0.1 0.0 0:07.56 httpd
10295 nagios 20 0 40976 1232 832 S 0.1 0.0 0:26.09 nrpe
19259 root 20 0 0 0 0 S 0.1 0.0 0:06.20 ldlm_poold
11 root RT 0 0 0 0 S 0.0 0.0 0:01.45 migration/2
21 root 20 0 0 0 0 S 0.0 0.0 0:14.56 ksoftirqd/4
90 root RT 0 0 0 0 S 0.0 0.0 0:00.56 watchdog/21
113 root 20 0 0 0 0 S 0.0 0.0 0:25.92 ksoftirqd/27
126 root RT 0 0 0 0 S 0.0 0.0 0:01.56 watchdog/30
131 root 20 0 0 0 0 S 0.0 0.0 0:18.65 events/0
132 root 20 0 0 0 0 S 0.0 0.0 0:05.54 events/1
133 root 20 0 0 0 0 S 0.0 0.0 0:12.65 events/2
139 root 20 0 0 0 0 S 0.0 0.0 0:12.88 events/8
140 root 20 0 0 0 0 S 0.0 0.0 0:05.66 events/9
143 root 20 0 0 0 0 S 0.0 0.0 0:12.99 events/12
144 root 20 0 0 0 0 S 0.0 0.0 0:04.73 events/13
145 root 20 0 0 0 0 S 0.0 0.0 0:13.54 events/14
146 root 20 0 0 0 0 S 0.0 0.0 0:05.12 events/15
147 root 20 0 0 0 0 S 0.0 0.0 0:16.32 events/16
148 root 20 0 0 0 0 S 0.0 0.0 0:07.37 events/17
149 root 20 0 0 0 0 S 0.0 0.0 0:12.68 events/18
151 root 20 0 0 0 0 S 0.0 0.0 0:12.18 events/20
152 root 20 0 0 0 0 S 0.0 0.0 0:05.03 events/21
154 root 20 0 0 0 0 S 0.0 0.0 0:05.06 events/23
156 root 20 0 0 0 0 S 0.0 0.0 0:04.66 events/25
158 root 20 0 0 0 0 S 0.0 0.0 0:04.97 events/27
159 root 20 0 0 0 0 S 0.0 0.0 0:12.13 events/28
160 root 20 0 0 0 0 S 0.0 0.0 0:05.09 events/29
162 root 20 0 0 0 0 S 0.0 0.0 0:05.47 events/31
1357 root 20 0 0 0 0 S 0.0 0.0 0:00.80 bdi-default
5172 root 20 0 0 0 0 S 0.0 0.0 0:07.65 poll_0_status
7138 root 20 0 0 0 0 S 0.0 0.0 0:03.96 ib_addr
7334 root 20 0 0 0 0 S 0.0 0.0 0:02.79 flush-253:1
7523 root 20 0 245m 2208 1064 S 0.0 0.0 0:08.47 rsyslogd
7792 root 20 0 22092 1044 808 S 0.0 0.0 0:12.85 xinetd
8379 root 20 0 151m 1292 1032 S 0.0 0.0 0:01.72 dsm_sa_eventmgr
10072 root 20 0 100m 18m 716 S 0.0 0.0 0:18.71 tslagentd
10511 root -2 0 36996 6312 2412 S 0.0 0.0 0:13.92 heartbeat
10536 nobody -2 0 34184 3500 2412 S 0.0 0.0 0:00.87 heartbeat
18741 root 20 0 0 0 0 S 0.0 0.0 0:03.77 ptlrpcd-brw
18742 root 20 0 0 0 0 S 0.0 0.0 0:11.14 ptlrpcd-brw-rcv
18743 root 20 0 0 0 0 S 0.0 0.0 0:09.17 ptlrpcd
18744 root 20 0 0 0 0 S 0.0 0.0 0:05.40 ptlrpcd-rcv
18916 root 20 0 0 0 0 S 0.0 0.0 0:02.65 flush-253:0
22172 root 20 0 4068 520 444 D 0.0 0.0 0:03.25 tee
1 root 20 0 19228 1560 1268 S 0.0 0.0 0:06.94 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 R 0.0 0.0 0:39.58 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:02.73 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:01.73 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:07.69 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
9 root 20 0 0 0 0 S 0.0 0.0 0:29.61 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/1
12 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/2
13 root 20 0 0 0 0 S 0.0 0.0 0:18.76 ksoftirqd/2
14 root RT 0 0 0 0 S 0.0 0.0 0:01.49 watchdog/2
15 root RT 0 0 0 0 S 0.0 0.0 0:05.00 migration/3
16 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/3
17 root 20 0 0 0 0 S 0.0 0.0 0:32.55 ksoftirqd/3
18 root RT 0 0 0 0 S 0.0 0.0 0:00.54 watchdog/3
19 root RT 0 0 0 0 S 0.0 0.0 0:01.06 migration/4
20 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/4
22 root RT 0 0 0 0 S 0.0 0.0 0:01.49 watchdog/4
23 root RT 0 0 0 0 S 0.0 0.0 0:04.46 migration/5
24 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/5
25 root 20 0 0 0 0 S 0.0 0.0 0:29.72 ksoftirqd/5
26 root RT 0 0 0 0 S 0.0 0.0 0:00.64 watchdog/5
27 root RT 0 0 0 0 S 0.0 0.0 0:00.96 migration/6
28 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/6
29 root 20 0 0 0 0 S 0.0 0.0 0:14.79 ksoftirqd/6
30 root RT 0 0 0 0 S 0.0 0.0 0:01.53 watchdog/6
31 root RT 0 0 0 0 S 0.0 0.0 0:03.63 migration/7
32 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/7
33 root 20 0 0 0 0 S 0.0 0.0 0:28.84 ksoftirqd/7
34 root RT 0 0 0 0 S 0.0 0.0 0:00.60 watchdog/7
35 root RT 0 0 0 0 S 0.0 0.0 0:00.46 migration/8
36 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/8
37 root 20 0 0 0 0 S 0.0 0.0 0:15.65 ksoftirqd/8
38 root RT 0 0 0 0 S 0.0 0.0 0:01.53 watchdog/8
39 root RT 0 0 0 0 S 0.0 0.0 0:03.10 migration/9
40 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/9
41 root 20 0 0 0 0 S 0.0 0.0 0:28.24 ksoftirqd/9
42 root RT 0 0 0 0 S 0.0 0.0 0:00.58 watchdog/9
43 root RT 0 0 0 0 S 0.0 0.0 0:00.68 migration/10
44 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/10
45 root 20 0 0 0 0 S 0.0 0.0 0:14.70 ksoftirqd/10
46 root RT 0 0 0 0 S 0.0 0.0 0:01.37 watchdog/10
47 root RT 0 0 0 0 S 0.0 0.0 0:02.29 migration/11
48 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/11
49 root 20 0 0 0 0 S 0.0 0.0 0:29.74 ksoftirqd/11
50 root RT 0 0 0 0 S 0.0 0.0 0:00.56 watchdog/11
51 root RT 0 0 0 0 S 0.0 0.0 0:00.45 migration/12
52 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/12
53 root 20 0 0 0 0 S 0.0 0.0 0:14.97 ksoftirqd/12
54 root RT 0 0 0 0 S 0.0 0.0 0:01.46 watchdog/12
55 root RT 0 0 0 0 S 0.0 0.0 0:01.81 migration/13
56 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/13
57 root 20 0 0 0 0 S 0.0 0.0 0:29.53 ksoftirqd/13
58 root RT 0 0 0 0 S 0.0 0.0 0:00.58 watchdog/13
59 root RT 0 0 0 0 S 0.0 0.0 0:00.32 migration/14
60 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/14
61 root 20 0 0 0 0 S 0.0 0.0 0:14.37 ksoftirqd/14
62 root RT 0 0 0 0 S 0.0 0.0 0:01.63 watchdog/14
63 root RT 0 0 0 0 S 0.0 0.0 0:01.73 migration/15
64 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/15
65 root 20 0 0 0 0 S 0.0 0.0 0:28.66 ksoftirqd/15
66 root RT 0 0 0 0 S 0.0 0.0 0:00.62 watchdog/15
67 root RT 0 0 0 0 S 0.0 0.0 0:00.15 migration/16
68 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/16
69 root 20 0 0 0 0 S 0.0 0.0 0:05.34 ksoftirqd/16
70 root RT 0 0 0 0 S 0.0 0.0 0:01.57 watchdog/16
71 root RT 0 0 0 0 S 0.0 0.0 0:00.27 migration/17
72 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/17
73 root 20 0 0 0 0 S 0.0 0.0 0:27.63 ksoftirqd/17
74 root RT 0 0 0 0 S 0.0 0.0 0:00.55 watchdog/17
75 root RT 0 0 0 0 S 0.0 0.0 0:01.97 migration/18
76 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/18
77 root 20 0 0 0 0 S 0.0 0.0 0:15.05 ksoftirqd/18
78 root RT 0 0 0 0 S 0.0 0.0 0:01.36 watchdog/18
79 root RT 0 0 0 0 S 0.0 0.0 0:01.13 migration/19
80 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/19
81 root 20 0 0 0 0 S 0.0 0.0 0:28.45 ksoftirqd/19
82 root RT 0 0 0 0 S 0.0 0.0 0:00.37 watchdog/19
83 root RT 0 0 0 0 S 0.0 0.0 0:00.42 migration/20
84 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/20
85 root 20 0 0 0 0 S 0.0 0.0 0:14.72 ksoftirqd/20
86 root RT 0 0 0 0 S 0.0 0.0 0:01.59 watchdog/20
87 root RT 0 0 0 0 S 0.0 0.0 0:00.42 migration/21
88 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/21
89 root 20 0 0 0 0 S 0.0 0.0 0:27.33 ksoftirqd/21
91 root RT 0 0 0 0 S 0.0 0.0 0:00.13 migration/22
92 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/22
93 root 20 0 0 0 0 S 0.0 0.0 0:16.21 ksoftirqd/22
94 root RT 0 0 0 0 S 0.0 0.0 0:20.14 watchdog/22
95 root RT 0 0 0 0 S 0.0 0.0 0:00.29 migration/23
96 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/23
97 root 20 0 0 0 0 S 0.0 0.0 0:27.58 ksoftirqd/23
98 root RT 0 0 0 0 S 0.0 0.0 0:00.56 watchdog/23
99 root RT 0 0 0 0 S 0.0 0.0 0:00.06 migration/24
100 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/24
101 root 20 0 0 0 0 S 0.0 0.0 0:14.39 ksoftirqd/24
102 root RT 0 0 0 0 S 0.0 0.0 0:01.58 watchdog/24
103 root RT 0 0 0 0 S 0.0 0.0 0:00.17 migration/25
104 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/25
105 root 20 0 0 0 0 S 0.0 0.0 0:26.51 ksoftirqd/25
106 root RT 0 0 0 0 S 0.0 0.0 0:00.58 watchdog/25
107 root RT 0 0 0 0 S 0.0 0.0 0:00.05 migration/26
108 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/26
109 root 20 0 0 0 0 S 0.0 0.0 0:15.02 ksoftirqd/26
110 root RT 0 0 0 0 S 0.0 0.0 0:01.60 watchdog/26
111 root RT 0 0 0 0 S 0.0 0.0 0:00.14 migration/27
112 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/27
114 root RT 0 0 0 0 S 0.0 0.0 0:00.59 watchdog/27
115 root RT 0 0 0 0 S 0.0 0.0 0:00.02 migration/28
116 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/28
117 root 20 0 0 0 0 S 0.0 0.0 0:13.59 ksoftirqd/28
118 root RT 0 0 0 0 S 0.0 0.0 0:01.49 watchdog/28
119 root RT 0 0 0 0 S 0.0 0.0 0:00.22 migration/29
120 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/29
121 root 20 0 0 0 0 S 0.0 0.0 0:24.67 ksoftirqd/29
122 root RT 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/29
123 root RT 0 0 0 0 S 0.0 0.0 0:00.01 migration/30
124 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/30
125 root 20 0 0 0 0 S 0.0 0.0 0:13.85 ksoftirqd/30
127 root RT 0 0 0 0 S 0.0 0.0 0:00.20 migration/31
128 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/31
129 root 20 0 0 0 0 S 0.0 0.0 0:25.39 ksoftirqd/31
130 root RT 0 0 0 0 S 0.0 0.0 0:00.61 watchdog/31
155 root 20 0 0 0 0 S 0.0 0.0 0:12.19 events/24
163 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cgroup
164 root 20 0 0 0 0 S 0.0 0.0 0:00.11 khelper
1194 root 20 0 0 0 0 S 0.0 0.0 0:00.00 async/mgr
1195 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pm
1355 root 20 0 0 0 0 S 0.0 0.0 0:02.38 sync_supers
1358 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/0
1359 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/1
1360 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/2
1361 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/3
1362 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/4
1363 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/5
1364 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/6
1365 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/7
1366 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/8
1367 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/9
1368 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/10
1369 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/11
1370 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/12
1371 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/13
1372 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/14
1373 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/15
1374 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/16
1375 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/17
1376 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/18
1377 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/19
1378 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/20
1379 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/21
1380 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/22
1381 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/23
1382 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/24
1383 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/25
1384 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/26
1385 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/27
1386 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/28
1387 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/29
1388 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/30
1389 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/31
1391 root 20 0 0 0 0 S 0.0 0.0 0:00.35 kblockd/0
1392 root 20 0 0 0 0 S 0.0 0.0 0:00.21 kblockd/1
1393 root 20 0 0 0 0 S 0.0 0.0 0:00.03 kblockd/2
1394 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kblockd/3
1395 root 20 0 0 0 0 S 0.0 0.0 0:00.06 kblockd/4
1396 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kblockd/5
1397 root 20 0 0 0 0 S 0.0 0.0 0:00.10 kblockd/6
1398 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kblockd/7
1399 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kblockd/8
1400 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kblockd/9
1401 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kblockd/10
1402 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kblockd/11
1403 root 20 0 0 0 0 S 0.0 0.0 0:00.20 kblockd/12
1404 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kblockd/13
1405 root 20 0 0 0 0 S 0.0 0.0 0:00.22 kblockd/14
1406 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kblockd/15
1407 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kblockd/16
1408 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kblockd/17
1409 root 20 0 0 0 0 S 0.0 0.0 0:00.03 kblockd/18
1410 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kblockd/19
1411 root 20 0 0 0 0 S 0.0 0.0 0:00.04 kblockd/20
1412 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kblockd/21
1413 root 20 0 0 0 0 S 0.0 0.0 0:00.03 kblockd/22
1414 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kblockd/23
1415 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kblockd/24
1416 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kblockd/25
1417 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kblockd/26
1418 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kblockd/27
1419 root 20 0 0 0 0 S 0.0 0.0 0:00.51 kblockd/28
1420 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kblockd/29
1421 root 20 0 0 0 0 S 0.0 0.0 0:00.35 kblockd/30
1422 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kblockd/31
1424 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpid
1425 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpi_notify
1426 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpi_hotplug
1688 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/0
1689 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/1
1690 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/2
1691 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/3
1692 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/4
1693 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/5
1694 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/6
1695 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/7
1696 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/8
1697 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/9
1698 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/10
1699 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/11
1700 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/12
1701 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/13
1702 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/14
1703 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/15
1704 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/16
1705 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/17
1706 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/18
1707 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/19
1708 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/20
1709 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/21
1710 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/22
1711 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/23
1712 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/24
1713 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/25
1714 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/26
1715 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/27
1716 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/28
1717 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/29
1718 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/30
1719 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/31
1720 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata_aux
1724 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksuspend_usbd
1729 root 20 0 0 0 0 S 0.0 0.0 0:00.13 khubd
1732 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kseriod
1736 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/0
1737 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/1
1738 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/2
1739 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/3
1740 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/4
1741 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/5
1742 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/6
1743 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/7
1744 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/8
1745 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/9
1746 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/10
1747 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/11
1748 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/12
1749 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/13
1750 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/14
1751 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/15
1752 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/16
1753 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/17
1754 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/18
1755 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/19
1756 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/20
1757 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/21
1758 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/22
1759 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/23
1760 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/24
1761 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/25
1762 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/26
1763 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/27
1764 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/28
1765 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/29
1766 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/30
1767 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/31
1768 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/0
1769 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/1
1770 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/2
1771 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/3
1772 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/4
1773 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/5
1774 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/6
1775 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/7
1776 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/8
1777 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/9
1778 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/10
1779 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/11
1780 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/12
1781 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/13
1782 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/14
1783 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/15
1784 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/16
1785 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/17
1786 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/18
1787 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/19
1788 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/20
1789 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/21
1790 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/22
1791 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/23
1792 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/24
1793 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/25
1794 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/26
1795 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/27
1796 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/28
1797 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/29
1798 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/30
1799 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/31
1804 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kslowd000
1805 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kslowd001
1837 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/0
1838 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/1
1839 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/2
1840 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/3
1841 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/4
1842 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/5
1843 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/6
1844 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/7
1845 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/8
1846 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/9
1847 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/10
1848 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/11
1849 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/12
1850 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/13
1851 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/14
1852 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/15
1853 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/16
1854 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/17
1855 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/18
1856 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/19
1857 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/20
1858 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/21
1859 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/22
1860 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/23
1861 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/24
1862 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/25
1863 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/26
1864 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/27
1865 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/28
1866 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/29
1867 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/30
1868 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rpciod/31
2142 root 20 0 0 0 0 S 0.0 0.0 0:00.37 khungtaskd
2147 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0
2148 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd1
2149 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
2150 root 39 19 0 0 0 S 0.0 0.0 0:00.28 khugepaged
2151 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/0
2152 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/1
2153 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/2
2154 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/3
2155 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/4
2156 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/5
2157 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/6
2158 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/7
2159 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/8
2160 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/9
2161 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/10
2162 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/11
2163 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/12
2164 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/13
2165 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/14
2166 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/15
2167 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/16
2168 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/17
2169 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/18
2170 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/19
2171 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/20
2172 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/21
2173 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/22
2174 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/23
2175 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/24
2176 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/25
2177 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/26
2178 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/27
2179 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/28
2180 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/29
2181 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/30
2182 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/31
2183 root 20 0 0 0 0 S 0.0 0.0 0:00.00 nfsiod
2184 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/0
2185 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/1
2186 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/2
2187 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/3
2188 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/4
2189 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/5
2190 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/6
2191 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/7
2192 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/8
2193 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/9
2194 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/10
2195 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/11
2196 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/12
2197 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/13
2198 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/14
2199 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/15
2200 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/16
2201 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/17
2202 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/18
2203 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/19
2204 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/20
2205 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/21
2206 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/22
2207 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/23
2208 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/24
2209 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/25
2210 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/26
2211 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/27
2212 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/28
2213 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/29
2214 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/30
2215 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/31
2912 root 20 0 0 0 0 S 0.0 0.0 0:00.00 iscsi_eh
2984 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kpsmoused
2991 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kstriped
2993 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/0
2994 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/1
2995 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/2
2996 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/3
2997 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/4
2998 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/5
2999 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/6
3000 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/7
3001 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/8
3002 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/9
3003 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/10
3004 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/11
3005 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/12
3006 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/13
3007 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/14
3008 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/15
3009 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/16
3010 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/17
3011 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/18
3012 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/19
3013 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/20
3014 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/21
3015 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/22
3016 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/23
3017 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/24
3018 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/25
3019 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/26
3020 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/27
3021 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/28
3022 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/29
3023 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/30
3024 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/31
3025 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpath_handlerd
3807 root 20 0 0 0 0 S 0.0 0.0 0:21.23 ll_mgs_04
4727 root 20 0 11332 1172 1020 S 0.0 0.0 0:00.00 sh
4728 root 20 0 16780 748 608 D 0.0 0.0 0:00.00 perfquery
4745 root 20 0 9232 596 404 D 0.0 0.0 0:00.00 ts-slow-ping-te
4746 root 20 0 9232 596 404 D 0.0 0.0 0:00.00 ts-slow-ping-te
4748 root 20 0 9232 596 404 D 0.0 0.0 0:00.00 ts-slow-ping-te
4750 root 20 0 9232 596 404 D 0.0 0.0 0:00.00 ts-slow-ping-te
4752 root 20 0 9232 596 404 D 0.0 0.0 0:00.00 ts-slow-ping-te
4759 root 20 0 3924 292 232 S 0.0 0.0 0:00.00 time
4760 root 20 0 16776 700 580 D 0.0 0.0 0:00.00 ibportstate
5088 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
5109 root 20 0 0 0 0 S 0.0 0.0 0:02.05 mlx4
5115 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mlx4_en
5129 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1
5134 root 20 0 0 0 0 S 0.0 0.0 0:00.00 fw_event0
5234 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdmflush
5240 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpath_rdacd
5277 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdmflush
5282 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdmflush
5287 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdmflush
5292 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdmflush
5325 root 20 0 0 0 0 D 0.0 0.0 0:03.06 kjournald
5406 root 16 -4 11304 1336 312 S 0.0 0.0 0:00.11 udevd
6890 root 20 0 0 0 0 S 0.0 0.0 0:00.47 kauditd
6936 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kjournald
6937 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kjournald
7093 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/0
7094 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/1
7095 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/2
7096 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/3
7097 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/4
7098 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/5
7099 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/6
7100 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/7
7101 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/8
7102 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/9
7103 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/10
7104 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/11
7105 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/12
7106 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/13
7107 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/14
7108 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/15
7109 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/16
7110 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/17
7111 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/18
7112 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/19
7113 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/20
7114 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/21
7115 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/22
7116 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/23
7117 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/24
7118 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/25
7119 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/26
7120 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/27
7121 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/28
7122 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/29
7123 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/30
7124 root 20 0 0 0 0 S 0.0 0.0 0:00.00 infiniband/31
7125 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mlx4_ib
7134 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_mcast
7142 root 20 0 0 0 0 S 0.0 0.0 0:00.00 iw_cm_wq
7146 root 20 0 0 0 0 S 0.0 0.0 0:00.12 ib_cm/0
7147 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/1
7148 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ib_cm/2
7149 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/3
7150 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/4
7151 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/5
7152 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/6
7153 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/7
7154 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/8
7155 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/9
7156 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/10
7157 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/11
7158 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/12
7159 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/13
7160 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/14
7161 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/15
7162 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/16
7163 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/17
7164 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/18
7165 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/19
7166 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/20
7167 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/21
7168 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/22
7169 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/23
7170 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/24
7171 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/25
7172 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/26
7173 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/27
7174 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/28
7175 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/29
7176 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/30
7177 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_cm/31
7181 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rdma_cm
7197 root 20 0 0 0 0 S 0.0 0.0 0:04.27 ipoib
7204 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ib_fmr(mlx4_0)
7498 root 16 -4 93172 856 592 S 0.0 0.0 0:02.00 auditd
7516 root 20 0 6156 600 508 S 0.0 0.0 0:00.00 portreserve
7557 rpc 20 0 19036 904 668 S 0.0 0.0 0:00.43 rpcbind
7577 rpcuser 20 0 23344 1332 900 S 0.0 0.0 0:00.00 rpc.statd
7607 root 20 0 23088 520 320 S 0.0 0.0 0:00.00 rpc.idmapd
7628 root 20 0 13548 708 564 S 0.0 0.0 0:13.79 lldpad
7652 dbus 20 0 21540 992 664 S 0.0 0.0 0:00.00 dbus-daemon
7663 avahi 20 0 27792 1504 1220 S 0.0 0.0 0:00.15 avahi-daemon
7664 avahi 20 0 27668 508 336 S 0.0 0.0 0:00.00 avahi-daemon
7692 root 20 0 4076 636 524 S 0.0 0.0 0:00.00 acpid
7701 haldaemo 20 0 25764 4552 3156 S 0.0 0.0 0:00.97 hald
7702 root 20 0 18104 1156 956 S 0.0 0.0 0:00.00 hald-runner
7727 root 20 0 20220 1076 924 S 0.0 0.0 0:00.00 hald-addon-inpu
7736 haldaemo 20 0 17804 1036 888 S 0.0 0.0 0:00.00 hald-addon-acpi
7753 root 20 0 94828 836 652 S 0.0 0.0 0:04.87 tsagentd
7769 root 20 0 6244 292 180 S 0.0 0.0 0:00.00 mcelog
7782 root 20 0 64112 1204 536 S 0.0 0.0 0:00.78 sshd
7830 ntp 20 0 25936 1488 1068 S 0.0 0.0 0:00.33 ntpd
7842 uuidd 20 0 6120 200 120 S 0.0 0.0 0:00.00 uuidd
7867 root 20 0 20376 1240 644 S 0.0 0.0 0:00.29 crond
7878 root 20 0 21452 484 300 S 0.0 0.0 0:00.00 atd
7931 apache 20 0 267m 5804 640 S 0.0 0.0 0:00.00 httpd
7932 apache 20 0 267m 5804 640 S 0.0 0.0 0:00.00 httpd
7933 apache 20 0 267m 5804 640 S 0.0 0.0 0:00.00 httpd
7934 apache 20 0 267m 5804 640 S 0.0 0.0 0:00.00 httpd
7935 apache 20 0 267m 5804 640 S 0.0 0.0 0:00.00 httpd
7936 apache 20 0 267m 5804 640 S 0.0 0.0 0:00.00 httpd
7937 apache 20 0 267m 5804 640 S 0.0 0.0 0:00.00 httpd
7938 apache 20 0 267m 5804 640 S 0.0 0.0 0:00.00 httpd
8463 root 20 0 17784 848 552 S 0.0 0.0 0:02.09 tsdiskd
8672 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 sh <defunct>
8734 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 sh <defunct>
8770 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 sh <defunct>
8806 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 sh <defunct>
9339 root 20 0 0 0 0 S 0.0 0.0 0:00.00 usbhid_resumer
9384 root 20 0 4060 536 468 S 0.0 0.0 0:00.00 mingetty
9386 root 20 0 4060 540 468 S 0.0 0.0 0:00.00 mingetty
9388 root 20 0 4060 540 468 S 0.0 0.0 0:00.00 mingetty
9390 root 20 0 4060 540 468 S 0.0 0.0 0:00.00 mingetty
9392 root 20 0 4060 536 468 S 0.0 0.0 0:00.00 mingetty
9394 root 20 0 4060 536 468 S 0.0 0.0 0:00.00 mingetty
9433 root 18 -2 11300 1372 352 S 0.0 0.0 0:00.00 udevd
9456 root 18 -2 11300 1352 332 S 0.0 0.0 0:00.00 udevd
10530 nobody -2 0 34188 3504 2412 S 0.0 0.0 0:00.00 heartbeat
10531 nobody -2 0 34184 3500 2412 S 0.0 0.0 0:02.07 heartbeat
10532 nobody -2 0 34184 3500 2412 S 0.0 0.0 0:00.84 heartbeat
10533 nobody -2 0 34184 3500 2412 S 0.0 0.0 0:02.25 heartbeat
10534 nobody -2 0 34184 3500 2412 S 0.0 0.0 0:00.89 heartbeat
10535 nobody -2 0 34184 3500 2412 S 0.0 0.0 0:01.07 heartbeat
10746 root 20 0 9232 1232 1044 S 0.0 0.0 0:07.02 ts-slow-ping-te
13212 root 20 0 0 0 0 S 0.0 0.0 10:48.13 mdt_88
13294 root 20 0 11580 1760 1344 S 0.0 0.0 0:00.00 bash
13785 root 20 0 0 0 0 S 0.0 0.0 10:42.87 mdt_91
13976 root 20 0 0 0 0 S 0.0 0.0 10:38.44 mdt_92
15386 root 20 0 0 0 0 S 0.0 0.0 10:42.97 mdt_93
17118 root 20 0 0 0 0 S 0.0 0.0 0:17.76 ldlm_cn_04
17121 root 20 0 0 0 0 S 0.0 0.0 0:08.48 ldlm_cn_05
17123 root 20 0 0 0 0 S 0.0 0.0 0:06.93 ldlm_cn_06
17124 root 20 0 0 0 0 S 0.0 0.0 0:08.56 ldlm_cn_07
17254 root 20 0 0 0 0 S 0.0 0.0 0:14.24 ldlm_cn_08
17255 root 20 0 0 0 0 S 0.0 0.0 0:08.02 ldlm_cn_09
17256 root 20 0 0 0 0 S 0.0 0.0 0:08.79 ldlm_cn_10
17257 root 20 0 0 0 0 S 0.0 0.0 0:07.79 ldlm_cn_11
17258 root 20 0 0 0 0 S 0.0 0.0 0:11.92 ldlm_cn_12
17259 root 20 0 0 0 0 S 0.0 0.0 0:16.98 ldlm_cn_13
17260 root 20 0 0 0 0 S 0.0 0.0 0:06.00 ldlm_cn_14
17261 root 20 0 0 0 0 S 0.0 0.0 0:10.84 ldlm_cn_15
17262 root 20 0 0 0 0 S 0.0 0.0 0:08.84 ldlm_cn_16
17263 root 20 0 0 0 0 S 0.0 0.0 0:05.07 ldlm_cn_17
17264 root 20 0 0 0 0 S 0.0 0.0 0:06.51 ldlm_cn_18
17265 root 20 0 0 0 0 S 0.0 0.0 0:08.06 ldlm_cn_19
17266 root 20 0 0 0 0 S 0.0 0.0 0:05.54 ldlm_cn_20
17284 root 20 0 0 0 0 S 0.0 0.0 0:15.03 ldlm_cn_22
17285 root 20 0 0 0 0 S 0.0 0.0 0:09.44 ldlm_cn_23
18638 root 20 0 0 0 0 S 0.0 0.0 0:00.10 cfs_wi_sd000
18639 root 20 0 0 0 0 S 0.0 0.0 0:00.07 cfs_wi_sd001
18640 root 20 0 0 0 0 S 0.0 0.0 0:00.09 cfs_wi_sd002
18641 root 20 0 0 0 0 S 0.0 0.0 0:00.12 cfs_wi_sd003
18642 root 20 0 0 0 0 S 0.0 0.0 0:00.10 cfs_wi_sd004
18643 root 20 0 0 0 0 S 0.0 0.0 0:00.11 cfs_wi_sd005
18644 root 20 0 0 0 0 S 0.0 0.0 0:00.09 cfs_wi_sd006
18645 root 20 0 0 0 0 S 0.0 0.0 0:00.13 cfs_wi_sd007
18646 root 20 0 0 0 0 S 0.0 0.0 0:00.09 cfs_wi_sd008
18647 root 20 0 0 0 0 S 0.0 0.0 0:00.11 cfs_wi_sd009
18648 root 20 0 0 0 0 S 0.0 0.0 0:00.09 cfs_wi_sd010
18649 root 20 0 0 0 0 S 0.0 0.0 0:00.10 cfs_wi_sd011
18650 root 20 0 0 0 0 S 0.0 0.0 0:00.10 cfs_wi_sd012
18651 root 20 0 0 0 0 S 0.0 0.0 0:00.07 cfs_wi_sd013
18652 root 20 0 0 0 0 S 0.0 0.0 0:00.08 cfs_wi_sd014
18653 root 20 0 0 0 0 S 0.0 0.0 0:00.08 cfs_wi_sd015
18654 root 20 0 0 0 0 S 0.0 0.0 0:00.08 cfs_wi_sd016
18655 root 20 0 0 0 0 S 0.0 0.0 0:00.10 cfs_wi_sd017
18656 root 20 0 0 0 0 S 0.0 0.0 0:00.10 cfs_wi_sd018
18657 root 20 0 0 0 0 S 0.0 0.0 0:00.12 cfs_wi_sd019
18658 root 20 0 0 0 0 S 0.0 0.0 0:00.08 cfs_wi_sd020
18659 root 20 0 0 0 0 S 0.0 0.0 0:00.09 cfs_wi_sd021
18660 root 20 0 0 0 0 S 0.0 0.0 0:00.09 cfs_wi_sd022
18661 root 20 0 0 0 0 S 0.0 0.0 0:00.10 cfs_wi_sd023
18662 root 20 0 0 0 0 S 0.0 0.0 0:00.08 cfs_wi_sd024
18663 root 20 0 0 0 0 S 0.0 0.0 0:00.09 cfs_wi_sd025
18664 root 20 0 0 0 0 S 0.0 0.0 0:00.10 cfs_wi_sd026
18665 root 20 0 0 0 0 S 0.0 0.0 0:00.12 cfs_wi_sd027
18666 root 20 0 0 0 0 S 0.0 0.0 0:00.12 cfs_wi_sd028
18667 root 20 0 0 0 0 S 0.0 0.0 0:00.08 cfs_wi_sd029
18668 root 20 0 0 0 0 S 0.0 0.0 0:00.07 cfs_wi_sd030
18669 root 20 0 0 0 0 S 0.0 0.0 0:00.09 cfs_wi_sd031
18670 root 20 0 0 0 0 S 0.0 0.0 0:00.00 wi_serial_sd
18671 root 20 0 0 0 0 S 0.0 0.0 0:00.00 obd_zombid
18672 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_0
18673 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_1
18674 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_2
18675 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_3
18676 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_4
18677 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_5
18678 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ptlrpc_hr_6
18679 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_7
18680 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_8
18681 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_9
18682 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_10
18683 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_11
18684 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ptlrpc_hr_12
18685 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_13
18686 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_14
18687 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_15
18688 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_16
18689 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_17
18690 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_18
18691 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_19
18692 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ptlrpc_hr_20
18693 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_21
18694 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ptlrpc_hr_22
18695 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_23
18696 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_24
18697 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_25
18698 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ptlrpc_hr_26
18699 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_27
18700 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_28
18701 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_29
18702 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_30
18703 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ptlrpc_hr_31
18740 root 20 0 0 0 0 S 0.0 0.0 0:03.70 kiblnd_connd
18745 root 20 0 0 0 0 S 0.0 0.0 0:00.40 ll_ping
18748 root 20 0 0 0 0 S 0.0 0.0 0:00.01 sptlrpc_gc
18762 root 20 0 0 0 0 S 0.0 0.0 10:33.74 mdt_95
18826 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ll_capa
19136 root 20 0 0 0 0 D 0.0 0.0 0:01.73 kmmpd-dm-0
19137 root 20 0 0 0 0 S 0.0 0.0 0:00.45 jbd2/dm-0-8
19138 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19139 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19140 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19141 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19142 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19143 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19144 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19145 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19146 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19147 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19148 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19149 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19150 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19151 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19152 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19153 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19154 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19155 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19156 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19157 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19158 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19159 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19160 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19161 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19162 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19163 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19164 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19165 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19166 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19167 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19168 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19169 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldiskfs-dio-unw
19251 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_bl_00
19252 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_bl_01
19253 root 20 0 0 0 0 S 0.0 0.0 0:10.32 ldlm_cn_00
19254 root 20 0 0 0 0 S 0.0 0.0 0:00.00 lc_watchdogd
19255 root 20 0 0 0 0 S 0.0 0.0 0:23.20 ldlm_cn_01
19256 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_00
19257 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_01
19258 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_elt
19260 root 20 0 0 0 0 S 0.0 0.0 0:23.81 ll_mgs_00
19263 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ll_evictor
19277 root 20 0 0 0 0 S 0.0 0.0 0:00.01 mdt_ck
19279 root 20 0 0 0 0 S 0.0 0.0 11:32.83 mdt_00
19280 root 20 0 0 0 0 S 0.0 0.0 12:35.59 mdt_01
19283 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mdt_attr_00
19284 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mdt_attr_01
19285 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mdt_mdsc_00
19286 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mdt_mdsc_01
19287 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mdt_mdss_00
19288 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mdt_mdss_01
19289 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mdt_dtss_00
19290 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mdt_dtss_01
19291 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mdt_fld_00
19292 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mdt_fld_01
19293 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mdt_xmds_00
19294 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mdt_xmds_01
19552 root 20 0 0 0 0 S 0.0 0.0 11:19.85 mdt_02
19558 root 20 0 0 0 0 S 0.0 0.0 12:17.75 mdt_05
19568 root 20 0 0 0 0 S 0.0 0.0 11:19.71 mdt_06
19570 root 20 0 0 0 0 S 0.0 0.0 11:40.05 mdt_08
19571 root 20 0 0 0 0 S 0.0 0.0 11:28.90 mdt_09
19573 root 20 0 0 0 0 S 0.0 0.0 12:34.75 mdt_11
19576 root 20 0 0 0 0 S 0.0 0.0 12:06.26 mdt_14
19585 root 20 0 0 0 0 S 0.0 0.0 12:01.62 mdt_15
19588 root 20 0 0 0 0 S 0.0 0.0 11:11.84 mdt_16
19591 root 20 0 0 0 0 S 0.0 0.0 12:02.93 mdt_19
19592 root 20 0 0 0 0 S 0.0 0.0 11:04.16 mdt_20
19593 root 20 0 0 0 0 S 0.0 0.0 12:12.12 mdt_21
19595 root 20 0 0 0 0 S 0.0 0.0 11:26.11 mdt_23
19596 root 20 0 0 0 0 S 0.0 0.0 11:57.40 mdt_24
19597 root 20 0 0 0 0 S 0.0 0.0 11:14.89 mdt_25
19598 root 20 0 0 0 0 S 0.0 0.0 12:07.06 mdt_26
19600 root 20 0 0 0 0 S 0.0 0.0 12:09.13 mdt_28
19601 root 20 0 0 0 0 S 0.0 0.0 11:27.63 mdt_29
19603 root 20 0 0 0 0 S 0.0 0.0 11:14.78 mdt_31
19604 root 20 0 0 0 0 S 0.0 0.0 11:58.33 mdt_32
19605 root 20 0 0 0 0 S 0.0 0.0 11:22.24 mdt_33
19606 root 20 0 0 0 0 S 0.0 0.0 11:58.53 mdt_34
19608 root 20 0 0 0 0 S 0.0 0.0 11:21.42 mdt_36
19609 root 20 0 0 0 0 S 0.0 0.0 11:54.02 mdt_37
19672 root 20 0 0 0 0 S 0.0 0.0 11:21.22 mdt_38
19688 root 20 0 0 0 0 S 0.0 0.0 11:31.00 mdt_41
19689 root 20 0 0 0 0 S 0.0 0.0 11:38.01 mdt_42
19690 root 20 0 0 0 0 S 0.0 0.0 11:47.99 mdt_43
19691 root 20 0 0 0 0 S 0.0 0.0 0:21.73 ll_mgs_03
19694 root 20 0 0 0 0 S 0.0 0.0 11:20.44 mdt_44
19696 root 20 0 0 0 0 S 0.0 0.0 11:33.69 mdt_46
19697 root 20 0 0 0 0 S 0.0 0.0 11:29.52 mdt_47
19700 root 20 0 0 0 0 S 0.0 0.0 11:15.16 mdt_48
19701 root 20 0 0 0 0 S 0.0 0.0 12:18.90 mdt_49
19702 root 20 0 0 0 0 S 0.0 0.0 11:33.78 mdt_50
19714 root 20 0 0 0 0 S 0.0 0.0 11:24.37 mdt_58
19720 root 20 0 0 0 0 S 0.0 0.0 11:22.67 mdt_62
19721 root 20 0 0 0 0 S 0.0 0.0 11:32.81 mdt_63
19722 root 20 0 0 0 0 S 0.0 0.0 11:48.86 mdt_64
19726 root 20 0 0 0 0 S 0.0 0.0 11:41.85 mdt_68
19728 root 20 0 0 0 0 S 0.0 0.0 12:34.65 mdt_70
19730 root 20 0 0 0 0 S 0.0 0.0 11:18.72 mdt_72
19731 root 20 0 0 0 0 S 0.0 0.0 11:25.45 mdt_73
19733 root 20 0 0 0 0 S 0.0 0.0 11:45.83 mdt_75
19734 root 20 0 0 0 0 S 0.0 0.0 11:35.76 mdt_76
19735 root 20 0 0 0 0 S 0.0 0.0 11:23.39 mdt_77
19741 root 20 0 0 0 0 S 0.0 0.0 11:13.38 mdt_81
19742 root 20 0 0 0 0 S 0.0 0.0 12:37.16 mdt_82
19743 root 20 0 0 0 0 S 0.0 0.0 11:22.78 mdt_83
22425 root 20 0 0 0 0 S 0.0 0.0 0:12.21 ldlm_cn_02
22426 root 20 0 0 0 0 S 0.0 0.0 0:16.91 ldlm_cn_03
22475 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_02
22476 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_03
22477 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_04
22478 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_05
22479 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_06
22480 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_07
22481 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_08
22482 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_09
22483 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_10
22484 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_11
22485 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_12
22486 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_13
22487 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_14
22488 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_15
22489 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_16
22490 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_17
22491 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_18
22492 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_19
22493 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_20
22494 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_21
22495 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_22
22496 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_23
22497 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_24
22498 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_25
22499 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_26
22500 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_27
22501 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_28
22502 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_29
22503 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_30
22504 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_31
22505 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_32
22506 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_33
22507 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_34
22508 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_35
22509 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_36
22510 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_37
22511 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_38
22512 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_39
22513 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_40
22514 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_41
22515 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_42
22516 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_43
22517 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_44
22518 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_45
22519 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_46
22520 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_47
22521 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_48
22524 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ldlm_cb_49
22919 root 20 0 91664 3724 2840 S 0.0 0.0 0:00.00 sshd
22923 root 20 0 11472 1736 1312 S 0.0 0.0 0:00.00 bash
28049 root 20 0 9232 872 652 S 0.0 0.0 0:00.00 ts-network-test
28051 root 20 0 57728 3564 2812 S 0.0 0.0 0:00.00 ssh
47510 root 20 0 4017m 3476 1972 S 0.0 0.0 0:00.19 console-kit-dae
47673 root 20 0 63316 1928 1488 S 0.0 0.0 0:00.00 login

Comments

[Liang Zhen (Inactive)]

not sure about the reason yet, but one thing may be a little helpful is setting "options ko2iblnd schedulers=8" which will decrease kiblnd schedulers from 32 to 8.

[Roger Spellman (Inactive)]

> not sure about the reason yet, but one thing may be a little helpful is
> setting "options ko2iblnd schedulers=8" which will decrease kiblnd schedulers from 32 to 8.

Thanks. I will try that. Prior to the crash, I was running
nohup top -b -d 30 > top.log 2>&1

I noticed that the number of mdt_rdpg_ threads was 12 all day long. Then, about an hour before the crash, the number of mdt_rdpg threads went up quite rapidly. Near the crash, they hit 72.

Right now (on the peer MDS that currently has the MDT mounted), they are are 23. Might it help to set the max to 23 (or some other value), by writing a 23 to /mdt_readpage/threads_max ?

Also, an hour before the crash, the number of mdt threads went up from 85 to 95. Might it help to set mdt/threads_max to 85?

[Roger Spellman (Inactive)]

I have failed over, and I have verified that there are only 8 kiblnd threads.
I would like to know if changing the number of mdt_rdpg or mdt threads might be helpful. Please let me know.
Thanks.

[Roger Spellman (Inactive)]

Liang,
Adding "options ko2iblnd schedulers=8" on the MDSes seems to have resolved the issue. Thank you very much!

Should this be done on the OSSes as well?
What are the performance impacts for meta data and object data?

[Liang Zhen (Inactive)]

yes I think it's reasonable to decrease this on OSS as well, we actually have less threads by default in 2.3 and later releases, so performance is not an issue here.

[Roger Spellman (Inactive)]

> yes I think it's reasonable to decrease this on OSS as well, we actually have less
> threads by default in 2.3 and later releases, so performance is not an issue here.

Thanks. This option seems valuable for the 2.1.5 release. Is it OK to leave in that option for a 2.3 (or 2.4 or 2.5 release), or should I remove it for 2.3, so that we get the default behavior?

[John Fuchs-Chesney (Inactive)]

Am I OK to go ahead and call this resolved? Thanks.

[Roger Spellman (Inactive)]

Because there is a work around, I am OK with the issue being closed.

[John Fuchs-Chesney (Inactive)]

Workaround identified.