Details
-
Bug
-
Resolution: Fixed
-
Major
-
Lustre 2.6.0, Lustre 2.7.0, Lustre 2.5.3
-
3
-
9223372036854775807
Description
Recently we had a careless user setting the quota limits to a random large number on our production system. e.g.
lfs setquota -g y35 -i 5171717171717171609888 -I 5171717171717171905088 /short
MDS got stuck as a result of this:
2015-03-24 19:54:15 BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_042:30397]
2015-03-24 19:54:15 Modules linked in: osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic crc32c_intel libcfs(U) raid1 coretemp ib_srp(U) scsi_transport_srp(U) ext4 jbd2 mbcache sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf rdma_ucm(U) ib_ucm(U) mlx5_ib(U) mlx5_core(U) mlx4_en(U) knem(U) ipmi_devintf dm_round_robin microcode sg iTCO_wdt iTCO_vendor_support sd_mod sb_edac edac_core sr_mod cdrom lpc_ich mfd_core i2c_i801 isci libsas scsi_transport_sas wmi power_meter lpfc scsi_transport_fc scsi_tgt crc_t10dif megaraid_sas ahci dm_multipath dm_mod rdma_cm(U) iw_cm(U) ib_umad(U) ib_uverbs(U) ib_ipoib(U) ib_cm(U) mlx4_ib(U) ib_sa(U) mlx4_core(U) ib_mad(U) ib_core(U) ib_addr(U) ipv6 compat(U) igb i2c_algo_bit i2c_core ptp pps_core dca [last unloaded: scsi_wait_scan]
2015-03-24 19:54:16 CPU 5
2015-03-24 19:54:16 Modules linked in: osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic crc32c_intel libcfs(U) raid1 coretemp ib_srp(U) scsi_transport_srp(U) ext4 jbd2 mbcache sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf rdma_ucm(U) ib_ucm(U) mlx5_ib(U) mlx5_core(U) mlx4_en(U) knem(U) ipmi_devintf dm_round_robin microcode sg iTCO_wdt iTCO_vendor_support sd_mod sb_edac edac_core sr_mod cdrom lpc_ich mfd_core i2c_i801 isci libsas scsi_transport_sas wmi power_meter lpfc scsi_transport_fc scsi_tgt crc_t10dif megaraid_sas ahci dm_multipath dm_mod rdma_cm(U) iw_cm(U) ib_umad(U) ib_uverbs(U) ib_ipoib(U) ib_cm(U) mlx4_ib(U) ib_sa(U) mlx4_core(U) ib_mad(U) ib_core(U) ib_addr(U) ipv6 compat(U) igb i2c_algo_bit i2c_core ptp pps_core dca [last unloaded: scsi_wait_scan]
2015-03-24 19:54:16
2015-03-24 19:54:16 Pid: 30397, comm: mdt01_042 Not tainted 2.6.32-431.23.3.el6_lustre.2.5.24.ddn2.x86_64 #1 FUJITSU PRIMERGY RX300 S7/D2939-A1
2015-03-24 19:54:16 RIP: 0010:[<ffffffffa0dbb730>] [<ffffffffa0dbb730>] qmt_adjust_qunit+0x3f0/0x660 [lquota]
2015-03-24 19:54:16 RSP: 0018:ffff883f385f5b90 EFLAGS: 00000286
2015-03-24 19:54:16 RAX: ffffffffffffffff RBX: ffff883f385f5bd0 RCX: 0000000000027b48
2015-03-24 19:54:16 RDX: 0000000000027b48 RSI: 0000000000000006 RDI: 0000000000010000
2015-03-24 19:54:16 RBP: ffffffff8100bb8e R08: 0000000000000000 R09: 0000000000000001
2015-03-24 19:54:16 R10: 0000000000000000 R11: 0000000000000001 R12: ffff883f385f5b70
2015-03-24 19:54:16 R13: ffff881eb868ad40 R14: ffff881f3fb19cc0 R15: ffffffffa0e063ff
2015-03-24 19:54:16 FS: 0000000000000000(0000) GS:ffff880028340000(0000) knlGS:0000000000000000
2015-03-24 19:54:16 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
2015-03-24 19:54:16 CR2: 00007ff7eec03240 CR3: 000000404ed1b000 CR4: 00000000000407e0
2015-03-24 19:54:16 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
2015-03-24 19:54:16 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
2015-03-24 19:54:16 Process mdt01_042 (pid: 30397, threadinfo ffff883f385f4000, task ffff883f39ec2080)
2015-03-24 19:54:16 Stack:
2015-03-24 19:54:16 ffff883f385f5c28 00ff88400af11988 ffff883f385f5bd0 ffff88400af11918
2015-03-24 19:54:16 <d> ffff881f3f523000 ffff881eb868ad40 0000000000000000 ffff8815a51f3c80
2015-03-24 19:54:16 <d> ffff883f385f5c60 ffffffffa0db481b ffff880942278400 0000000000000002
2015-03-24 19:54:16 Call Trace:
2015-03-24 19:54:16 [<ffffffffa0db481b>] ? qmt_set.clone.0+0x52b/0x700 [lquota]
2015-03-24 19:54:16 [<ffffffffa0db4f78>] ? qmt_quotactl+0x588/0x610 [lquota]
2015-03-24 19:54:16 [<ffffffffa0f04ce4>] ? mdt_quotactl+0x574/0x780 [mdt]
2015-03-24 19:54:16 [<ffffffffa0f08aba>] ? mdt_handle_common+0x52a/0x1470 [mdt]
2015-03-24 19:54:16 [<ffffffffa0f45985>] ? mds_regular_handle+0x15/0x20 [mdt]
2015-03-24 19:54:16 [<ffffffffa08d2cf5>] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
2015-03-24 19:54:16 [<ffffffffa04e84ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
2015-03-24 19:54:16 [<ffffffffa04f93cf>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
2015-03-24 19:54:16 [<ffffffffa08ca3d9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
2015-03-24 19:54:16 [<ffffffff810546b9>] ? __wake_up_common+0x59/0x90
2015-03-24 19:54:16 [<ffffffffa08d405d>] ? ptlrpc_main+0xaed/0x1740 [ptlrpc]
2015-03-24 19:54:16 [<ffffffffa08d3570>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
2015-03-24 19:54:16 [<ffffffff8109abf6>] ? kthread+0x96/0xa0
2015-03-24 19:54:16 [<ffffffff8100c20a>] ? child_rip+0xa/0x20
2015-03-24 19:54:16 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
2015-03-24 19:54:16 [<ffffffff8100c200>] ? child_rip+0x0/0x20
2015-03-24 19:54:16 Code: 44 00 00 4d 8b 65 78 48 8b 4b 40 4d 63 f6 4b 8d 34 76 48 01 f6 48 89 f2 49 0f af d4 48 01 ca 48 39 d0 0f 82 53 fe ff ff 0f 1f 00 <49> c1 e4 02 4c 89 e2 48 0f af d6 48 01 ca 48 39 d0 73 ed e9 38
It got stuck in qmt_adjust_qunit, and couldn't call dt_trans_stop in qmt_set(), as a result: the jbd2 got stuck:
2015-03-24 19:55:12 INFO: task jbd2/dm-48-8:4135 blocked for more than 120 seconds.
2015-03-24 19:55:12 Not tainted 2.6.32-431.23.3.el6_lustre.2.5.24.ddn2.x86_64 #1
2015-03-24 19:55:12 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2015-03-24 19:55:12 jbd2/dm-48-8 D 0000000000000003 0 4135 2 0x00000000
2015-03-24 19:55:12 ffff881f56667d20 0000000000000046 0000000000016880 0000000000016880
2015-03-24 19:55:12 ffff882053648000 0000000000016880 0000000000016880 ffff881f5665d540
2015-03-24 19:55:12 ffff881f5665daf8 ffff881f56667fd8 000000000000fbc8 ffff881f5665daf8
2015-03-24 19:55:12 Call Trace:
2015-03-24 19:55:12 [<ffffffff8109b2ce>] ? prepare_to_wait+0x4e/0x80
2015-03-24 19:55:12 [<ffffffffa043b91f>] jbd2_journal_commit_transaction+0x19f/0x15a0 [jbd2]
2015-03-24 19:55:12 [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
2015-03-24 19:55:12 [<ffffffff81083e1c>] ? lock_timer_base+0x3c/0x70
2015-03-24 19:55:12 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
2015-03-24 19:55:12 [<ffffffffa0441c18>] kjournald2+0xb8/0x220 [jbd2]
2015-03-24 19:55:12 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
2015-03-24 19:55:12 [<ffffffffa0441b60>] ? kjournald2+0x0/0x220 [jbd2]
2015-03-24 19:55:12 [<ffffffff8109abf6>] kthread+0x96/0xa0
2015-03-24 19:55:12 [<ffffffff8100c20a>] child_rip+0xa/0x20
2015-03-24 19:55:12 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
2015-03-24 19:55:12 [<ffffffff8100c200>] ? child_rip+0x0/0x20
Then everyone got stuck trying to start a transaction, the filesystem was offline:
2015-03-24 19:55:12 INFO: task kswapd0:179 blocked for more than 120 seconds.
2015-03-24 19:55:12 Not tainted 2.6.32-431.23.3.el6_lustre.2.5.24.ddn2.x86_64 #1
2015-03-24 19:55:12 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2015-03-24 19:55:12 kswapd0 D 0000000000000003 0 179 2 0x00000000
2015-03-24 19:55:12 ffff88204ed1bb90 0000000000000046 0000000000000000 ffff88204ed1bb54
2015-03-24 19:55:12 ffff880200000000 ffff88207fc24800 ffff880028356880 0000000000000199
2015-03-24 19:55:12 ffff88204eca3058 ffff88204ed1bfd8 000000000000fbc8 ffff88204eca3058
2015-03-24 19:55:12 Call Trace:
2015-03-24 19:55:12 [<ffffffff8152a7de>] __mutex_lock_slowpath+0x13e/0x180
2015-03-24 19:55:12 [<ffffffff811a621a>] ? __destroy_inode+0x2a/0xa0
2015-03-24 19:55:12 [<ffffffff8152a67b>] mutex_lock+0x2b/0x50
2015-03-24 19:55:12 [<ffffffffa069c9bd>] lu_cache_shrink+0x4d/0x280 [obdclass]
2015-03-24 19:55:12 [<ffffffff811389aa>] shrink_slab+0x8a/0x1a0
2015-03-24 19:55:12 [<ffffffff8113bd6a>] balance_pgdat+0x59a/0x820
2015-03-24 19:55:12 [<ffffffff8113c124>] kswapd+0x134/0x3b0
2015-03-24 19:55:12 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
2015-03-24 19:55:12 [<ffffffff8113bff0>] ? kswapd+0x0/0x3b0
2015-03-24 19:55:12 [<ffffffff8109abf6>] kthread+0x96/0xa0
2015-03-24 19:55:12 [<ffffffff8100c20a>] child_rip+0xa/0x20
2015-03-24 19:55:12 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
2015-03-24 19:55:12 [<ffffffff8100c200>] ? child_rip+0x0/0x20
2015-03-24 19:55:12 INFO: task kswapd1:180 blocked for more than 120 seconds.
2015-03-24 19:55:12 Not tainted 2.6.32-431.23.3.el6_lustre.2.5.24.ddn2.x86_64 #1
2015-03-24 19:55:12 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2015-03-24 19:55:12 kswapd1 D 0000000000000008 0 180 2 0x00000000
2015-03-24 19:55:12 ffff88204ed1d7f0 0000000000000046 0000000000000000 ffff88204ed1d7b4
2015-03-24 19:55:12 0000000000000000 ffff88207fc25200 ffff880028356880 00000000000005fe
2015-03-24 19:55:12 ffff88204eca25f8 ffff88204ed1dfd8 000000000000fbc8 ffff88204eca25f8
2015-03-24 19:55:12 Call Trace:
2015-03-24 19:55:12 [<ffffffffa043a18a>] start_this_handle+0x27a/0x4a0 [jbd2]
2015-03-24 19:55:12 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
2015-03-24 19:55:12 [<ffffffffa043a5b0>] jbd2_journal_start+0xd0/0x110 [jbd2]
2015-03-24 19:55:12 [<ffffffffa0d4a746>] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs]
2015-03-24 19:55:12 [<ffffffffa0d4a943>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
2015-03-24 19:55:12 [<ffffffff811ecf8c>] dqget+0x2ac/0x390
2015-03-24 19:55:12 [<ffffffff811ed3da>] vfs_get_dqblk+0x1a/0xe0
2015-03-24 19:55:12 [<ffffffffa0e359d9>] osd_acct_index_lookup+0x219/0x2c0 [osd_ldiskfs]
2015-03-24 19:55:12 [<ffffffffa0d994e4>] lquota_disk_read+0xb4/0x360 [lquota]
2015-03-24 19:55:12 [<ffffffffa0da1a5a>] qsd_refresh_usage+0x7a/0x300 [lquota]
2015-03-24 19:55:12 [<ffffffffa0dae3e5>] qsd_op_adjust+0x2e5/0x580 [lquota]
2015-03-24 19:55:12 [<ffffffffa0e0e981>] osd_object_delete+0x231/0x2f0 [osd_ldiskfs]
2015-03-24 19:55:12 [<ffffffffa069b269>] lu_object_free+0x89/0x1b0 [obdclass]
2015-03-24 19:55:12 [<ffffffffa04fcb62>] ? cfs_hash_bd_from_key+0x42/0xd0 [libcfs]
2015-03-24 19:55:12 [<ffffffffa069c777>] lu_site_purge+0x2a7/0x4a0 [obdclass]
2015-03-24 19:55:12 [<ffffffffa069ca48>] lu_cache_shrink+0xd8/0x280 [obdclass]
2015-03-24 19:55:12 [<ffffffff81138a4a>] shrink_slab+0x12a/0x1a0
2015-03-24 19:55:12 [<ffffffff8113bd6a>] balance_pgdat+0x59a/0x820
2015-03-24 19:55:12 [<ffffffff8113c124>] kswapd+0x134/0x3b0
2015-03-24 19:55:12 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
2015-03-24 19:55:12 [<ffffffff8113bff0>] ? kswapd+0x0/0x3b0
2015-03-24 19:55:12 [<ffffffff8109abf6>] kthread+0x96/0xa0
2015-03-24 19:55:12 [<ffffffff8100c20a>] child_rip+0xa/0x20
2015-03-24 19:55:12 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
2015-03-24 19:55:12 [<ffffffff8100c200>] ? child_rip+0x0/0x20
2015-03-24 19:55:13 INFO: task mdt00_002:4249 blocked for more than 120 seconds.
2015-03-24 19:55:13 Not tainted 2.6.32-431.23.3.el6_lustre.2.5.24.ddn2.x86_64 #1
2015-03-24 19:55:13 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2015-03-24 19:55:13 mdt00_002 D 0000000000000000 0 4249 2 0x00000000
2015-03-24 19:55:13 ffff881f434838d0 0000000000000046 ffff881f43483870 ffffffffa0d5114e
2015-03-24 19:55:13 0000000000000000 0000000000000160 ffff88204df6e338 ffff88204df6e390
2015-03-24 19:55:13 ffff88204d8b25f8 ffff881f43483fd8 000000000000fbc8 ffff88204d8b25f8
2015-03-24 19:55:13 Call Trace:
2015-03-24 19:55:13 [<ffffffffa0d5114e>] ? ldiskfs_xattr_find_entry+0x3e/0x120 [ldiskfs]
2015-03-24 19:55:13 [<ffffffff8109b2ce>] ? prepare_to_wait+0x4e/0x80
2015-03-24 19:55:13 [<ffffffffa043a18a>] start_this_handle+0x27a/0x4a0 [jbd2]
2015-03-24 19:55:13 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
2015-03-24 19:55:13 [<ffffffffa043a5b0>] jbd2_journal_start+0xd0/0x110 [jbd2]
2015-03-24 19:55:13 [<ffffffffa0d4a746>] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs]
2015-03-24 19:55:13 [<ffffffffa0e0269f>] osd_trans_start+0x1df/0x660 [osd_ldiskfs]
2015-03-24 19:55:13 [<ffffffffa0d9995e>] ? lquota_disk_declare_write+0x1ce/0x390 [lquota]
2015-03-24 19:55:13 [<ffffffffa0dba1f9>] qmt_trans_start_with_slv+0x2b9/0x570 [lquota]
2015-03-24 19:55:13 [<ffffffffa0d9b4e6>] ? lquota_disk_slv_find+0x286/0x470 [lquota]
2015-03-24 19:55:13 [<ffffffffa0db29e9>] qmt_dqacq0+0x119/0xf30 [lquota]
2015-03-24 19:55:13 [<ffffffffa08e8ed6>] ? __req_capsule_get+0x166/0x710 [ptlrpc]
2015-03-24 19:55:13 [<ffffffffa08c2eae>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc]
2015-03-24 19:55:13 [<ffffffffa0db7148>] qmt_intent_policy+0x768/0x9e0 [lquota]
2015-03-24 19:55:13 [<ffffffffa0f03bb8>] mdt_intent_policy+0x548/0x770 [mdt]
2015-03-24 19:55:13 [<ffffffffa0879511>] ldlm_lock_enqueue+0x361/0x8c0 [ptlrpc]
2015-03-24 19:55:13 [<ffffffffa08a2aff>] ldlm_handle_enqueue0+0x4ef/0x10a0 [ptlrpc]
2015-03-24 19:55:13 [<ffffffffa0f03ee6>] mdt_enqueue+0x46/0xe0 [mdt]
2015-03-24 19:55:13 [<ffffffffa0f08aba>] mdt_handle_common+0x52a/0x1470 [mdt]
2015-03-24 19:55:13 [<ffffffffa0f45985>] mds_regular_handle+0x15/0x20 [mdt]
2015-03-24 19:55:13 [<ffffffffa08d2cf5>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
2015-03-24 19:55:13 [<ffffffffa04e84ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
2015-03-24 19:55:13 [<ffffffffa04f93cf>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
2015-03-24 19:55:13 [<ffffffffa08ca3d9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
2015-03-24 19:55:13 [<ffffffff810546b9>] ? __wake_up_common+0x59/0x90
2015-03-24 19:55:13 [<ffffffffa08d405d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
2015-03-24 19:55:13 [<ffffffffa08d3570>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
2015-03-24 19:55:13 [<ffffffff8109abf6>] kthread+0x96/0xa0
2015-03-24 19:55:13 [<ffffffff8100c20a>] child_rip+0xa/0x20
2015-03-24 19:55:13 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
2015-03-24 19:55:13 [<ffffffff8100c200>] ? child_rip+0x0/0x20
2015-03-24 19:56:12 LNet: Service thread pid 7082 was inactive for 200.02s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
2015-03-24 19:56:12 LNet: Skipped 1 previous similar message
2015-03-24 19:56:12 Pid: 7082, comm: mdt_rdpg02_019
2015-03-24 19:56:12
2015-03-24 19:56:12 Call Trace:
2015-03-24 19:56:12 [<ffffffffa043a18a>] start_this_handle+0x27a/0x4a0 [jbd2]
2015-03-24 19:56:12 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
2015-03-24 19:56:12 [<ffffffffa043a5b0>] jbd2_journal_start+0xd0/0x110 [jbd2]
2015-03-24 19:56:12 [<ffffffffa0f2d4a2>] ? mdt_txn_start_cb+0xe2/0x290 [mdt]
2015-03-24 19:56:12 [<ffffffffa0d4a746>] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs]
2015-03-24 19:56:12 [<ffffffffa0e0269f>] osd_trans_start+0x1df/0x660 [osd_ldiskfs]
2015-03-24 19:56:12 [<ffffffffa0fdf619>] lod_trans_start+0x1b9/0x250 [lod]
2015-03-24 19:56:12 [<ffffffffa106ce47>] mdd_trans_start+0x17/0x20 [mdd]
2015-03-24 19:56:12 [<ffffffffa1052633>] mdd_attr_set+0x4a3/0x1470 [mdd]
2015-03-24 19:56:12 [<ffffffffa0f3169c>] mdt_mfd_close+0x7ac/0x1bc0 [mdt]
2015-03-24 19:56:12 [<ffffffffa08c1e95>] ? lustre_msg_buf+0x55/0x60 [ptlrpc]
2015-03-24 19:56:12 [<ffffffffa08e8ed6>] ? __req_capsule_get+0x166/0x710 [ptlrpc]
2015-03-24 19:56:12 [<ffffffffa08c2eae>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc]
2015-03-24 19:56:13 [<ffffffffa06740f5>] ? class_handle2object+0x95/0x190 [obdclass]
2015-03-24 19:56:13 [<ffffffffa0f33e42>] mdt_close+0x642/0xa80 [mdt]
2015-03-24 19:56:13 [<ffffffffa0f08aba>] mdt_handle_common+0x52a/0x1470 [mdt]
2015-03-24 19:56:13 [<ffffffffa0f45965>] mds_readpage_handle+0x15/0x20 [mdt]
2015-03-24 19:56:13 [<ffffffffa08d2cf5>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
2015-03-24 19:56:13 [<ffffffffa04e84ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
2015-03-24 19:56:13 [<ffffffffa04f93cf>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
2015-03-24 19:56:13 [<ffffffffa08ca3d9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
2015-03-24 19:56:13 [<ffffffff810546b9>] ? __wake_up_common+0x59/0x90
2015-03-24 19:56:13 [<ffffffffa08d405d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
2015-03-24 19:56:13 [<ffffffffa08d3570>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
2015-03-24 19:56:13 [<ffffffff8109abf6>] kthread+0x96/0xa0
2015-03-24 19:56:13 [<ffffffff8100c20a>] child_rip+0xa/0x20
2015-03-24 19:56:13 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
2015-03-24 19:56:13 [<ffffffff8100c200>] ? child_rip+0x0/0x20