Details
-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
None
-
Lustre 2.5.0
-
None
-
Centos6 on megaraid perc hardware raid 6
-
3
-
12377
Description
We have had 4 or 5 oss's crash within the last few hours with similar issues. They have been running fine for over 30days, doing a massive backup (rsync) of another file system... and suddenly started crashing. Here are some syslogs
Jan 22 15:45:59 pdat0102 kernel: -----------[ cut here ]-----------
Jan 22 15:45:59 pdat0102 kernel: WARNING: at lib/list_debug.c:26 __list_add+0x6d/0xa0() (Not tainted)
Jan 22 15:45:59 pdat0102 kernel: Hardware name: empty
Jan 22 15:45:59 pdat0102 kernel: list_add corruption. next->prev should be prev (ffffc9002926b2e0), but was ffff881059861000. (next=ffff880aeb7813d0).
Jan 22 15:45:59 pdat0102 kernel: Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl osp(U) ofd(U) lfsck(U) ost(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U)
lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) mpt2sas scsi_transport_sas raid
_class mptctl mptbase autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bonding 8021q garp stp llc ipv6 sg ses enclosure igb ptp pps_core microcode i2c_i801 i2c_core se
rio_raw iTCO_wdt iTCO_vendor_support e1000e ioatdma dca i7core_edac edac_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif ahci megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [l
ast unloaded: scsi_wait_scan]
Jan 22 15:45:59 pdat0102 kernel: Pid: 3962, comm: ll_ost01_000 Not tainted 2.6.32-358.18.1.el6_lustre.x86_64 #1
Jan 22 15:45:59 pdat0102 kernel: Call Trace:
Jan 22 15:45:59 pdat0102 kernel: [<ffffffff8106e3e7>] ? warn_slowpath_common+0x87/0xc0
Jan 22 15:45:59 pdat0102 kernel: [<ffffffff8106e4d6>] ? warn_slowpath_fmt+0x46/0x50
Jan 22 15:45:59 pdat0102 kernel: [<ffffffffa03ded3b>] ? lnet_ni_send+0x4b/0xf0 [lnet]
Jan 22 15:45:59 pdat0102 kernel: [<ffffffff8128974d>] ? __list_add+0x6d/0xa0
Jan 22 15:45:59 pdat0102 kernel: [<ffffffffa03cf93b>] ? lnet_res_lh_initialize+0x4b/0x50 [lnet]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa03dacdc>] ? lnet_md_link+0x3c/0xe0 [lnet]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa03dba8f>] ? LNetMDBind+0x27f/0x4b0 [lnet]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa06057aa>] ? ptl_send_buf+0x12a/0x550 [ptlrpc]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa06253b8>] ? at_measured+0x108/0x380 [ptlrpc]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa06467d5>] ? null_authorize+0x75/0x100 [ptlrpc]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa0605e4b>] ? ptlrpc_send_reply+0x27b/0x7f0 [ptlrpc]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa060ce51>] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa05d48e4>] ? target_send_reply_msg+0x54/0x190 [ptlrpc]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa05d4e06>] ? target_send_reply+0x3e6/0x720 [ptlrpc]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa0c26325>] ? oti_to_request+0x75/0xc0 [ost]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa0c31433>] ? ost_handle+0x203/0x44d0 [ost]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa06149cb>] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa061ce25>] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa03374ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa034827f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa06144c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa061e18d>] ? ptlrpc_main+0xaed/0x1740 [ptlrpc]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffffa061d6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Jan 22 15:46:00 pdat0102 kernel: [<ffffffff81096a36>] ? kthread+0x96/0xa0
Jan 22 15:46:00 pdat0102 kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Jan 22 15:46:00 pdat0102 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 22 15:46:00 pdat0102 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 22 15:46:00 pdat0102 kernel: --[ end trace 35186327b68bc519 ]--
Jan 22 13:16:41 pdat0111 kernel: -----------[ cut here ]-----------
Jan 22 13:16:41 pdat0111 kernel: WARNING: at lib/list_debug.c:51 list_del+0x8d/0xa0() (Not tainted)
Jan 22 13:16:41 pdat0111 kernel: Hardware name: empty
Jan 22 13:16:41 pdat0111 kernel: list_del corruption. next->prev should be ffff8822f39e01c0, but was ffff88084557b570
Jan 22 13:16:42 pdat0111 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc(U) fid(U)
fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase autofs4 sunrpc c
pufreq_ondemand acpi_cpufreq freq_table mperf bonding 8021q garp stp llc ipv6 sg ses enclosure igb ptp pps_core microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support e
1000e ioatdma dca i7core_edac edac_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif megaraid_sas ahci dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Jan 22 13:16:42 pdat0111 kernel: Pid: 3622, comm: socknal_sd01_00 Not tainted 2.6.32-358.18.1.el6_lustre.x86_64 #1
Jan 22 13:16:42 pdat0111 kernel: Call Trace:
Jan 22 13:16:42 pdat0111 kernel: [<ffffffff8106e3e7>] ? warn_slowpath_common+0x87/0xc0
Jan 22 13:16:42 pdat0111 kernel: [<ffffffff8106e4d6>] ? warn_slowpath_fmt+0x46/0x50
Jan 22 13:16:42 pdat0111 kernel: [<ffffffff814b045a>] ? inet_recvmsg+0x5a/0x90
Jan 22 13:16:42 pdat0111 kernel: [<ffffffff812896cd>] ? list_del+0x8d/0xa0
Jan 22 13:16:42 pdat0111 kernel: [<ffffffffa03d76e4>] ? lnet_me_unlink+0x14/0x140 [lnet]
Jan 22 13:16:42 pdat0111 kernel: [<ffffffffa03db570>] ? lnet_md_unlink+0x250/0x340 [lnet]
Jan 22 13:16:42 pdat0111 kernel: [<ffffffffa03dcd3f>] ? lnet_try_match_md+0x22f/0x310 [lnet]
Jan 22 13:16:42 pdat0111 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 22 13:16:42 pdat0111 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 22 13:16:42 pdat0111 kernel: [<ffffffffa03dcebc>] ? lnet_mt_match_md+0x9c/0x1c0 [lnet]
Jan 22 13:16:42 pdat0111 kernel: [<ffffffffa03dd7c0>] ? lnet_ptl_match_md+0x280/0x870 [lnet]
Jan 22 13:16:42 pdat0111 kernel: [<ffffffffa03efb46>] ? lnet_nid2peer_locked+0x66/0x4b0 [lnet]
Jan 22 13:16:42 pdat0111 kernel: [<ffffffffa03e4f7b>] ? lnet_parse+0xb9b/0x18c0 [lnet]
Jan 22 13:16:42 pdat0111 kernel: [<ffffffffa07c5f24>] ? ksocknal_lib_recv_iov+0xe4/0x230 [ksocklnd]
Jan 22 13:16:42 pdat0111 kernel: [<ffffffffa07bd4b9>] ? ksocknal_recv_iov+0x29/0x130 [ksocklnd]
Jan 22 13:16:42 pdat0111 kernel: [<ffffffffa07be531>] ? ksocknal_process_receive+0x2b1/0xa00 [ksocklnd]
Jan 22 13:16:42 pdat0111 kernel: [<ffffffffa07c1145>] ? ksocknal_scheduler+0x105/0x760 [ksocklnd]
Jan 22 13:16:42 pdat0111 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 22 13:16:42 pdat0111 kernel: [<ffffffffa07c1040>] ? ksocknal_scheduler+0x0/0x760 [ksocklnd]
Jan 22 13:16:42 pdat0111 kernel: [<ffffffff81096a36>] ? kthread+0x96/0xa0
Jan 22 13:16:42 pdat0111 kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Jan 22 13:16:43 pdat0111 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 22 13:16:43 pdat0111 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 22 13:16:43 pdat0111 kernel: --[ end trace 22728380efe73915 ]--
Jan 22 15:45:17 pdat0103 kernel: -----------[ cut here ]-----------
Jan 22 15:45:17 pdat0103 kernel: WARNING: at lib/list_debug.c:48 list_del+0x6e/0xa0() (Not tainted)
Jan 22 15:45:17 pdat0103 kernel: Hardware name: empty
Jan 22 15:45:18 pdat0103 kernel: list_del corruption. prev->next should be ffff880a457a6ad0, but was 00000154005bf30a
Jan 22 15:45:18 pdat0103 kernel: Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl osp(U) ofd(U) lfsck(U) ost(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U)
lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) mpt2sas scsi_transport_sas raid
_class mptctl mptbase autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bonding 8021q garp stp llc ipv6 sg ses enclosure igb ptp pps_core microcode serio_raw i2c_i801 i
2c_core iTCO_wdt iTCO_vendor_support e1000e ioatdma dca i7core_edac edac_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif megaraid_sas ahci dm_mirror dm_region_hash dm_log dm_mod [l
ast unloaded: scsi_wait_scan]
Jan 22 15:45:18 pdat0103 kernel: Pid: 3913, comm: socknal_sd01_00 Not tainted 2.6.32-358.18.1.el6_lustre.x86_64 #1
Jan 22 15:45:18 pdat0103 kernel: Call Trace:
Jan 22 15:45:18 pdat0103 kernel: [<ffffffff8106e3e7>] ? warn_slowpath_common+0x87/0xc0
Jan 22 15:45:18 pdat0103 kernel: [<ffffffff8106e4d6>] ? warn_slowpath_fmt+0x46/0x50
Jan 22 15:45:18 pdat0103 kernel: [<ffffffff812896ae>] ? list_del+0x6e/0xa0
Jan 22 15:45:18 pdat0103 kernel: [<ffffffffa03db365>] ? lnet_md_unlink+0x45/0x340 [lnet]
Jan 22 15:45:18 pdat0103 kernel: [<ffffffffa03dcd3f>] ? lnet_try_match_md+0x22f/0x310 [lnet]
Jan 22 15:45:19 pdat0103 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 22 15:45:19 pdat0103 kernel: [<ffffffffa03dcebc>] ? lnet_mt_match_md+0x9c/0x1c0 [lnet]
Jan 22 15:45:19 pdat0103 kernel: [<ffffffffa03dd7c0>] ? lnet_ptl_match_md+0x280/0x870 [lnet]
Jan 22 15:45:19 pdat0103 kernel: [<ffffffffa03efb46>] ? lnet_nid2peer_locked+0x66/0x4b0 [lnet]
Jan 22 15:45:19 pdat0103 kernel: [<ffffffffa03e4f7b>] ? lnet_parse+0xb9b/0x18c0 [lnet]
Jan 22 15:45:19 pdat0103 kernel: [<ffffffffa07daf24>] ? ksocknal_lib_recv_iov+0xe4/0x230 [ksocklnd]
Jan 22 15:45:19 pdat0103 kernel: [<ffffffffa07d24b9>] ? ksocknal_recv_iov+0x29/0x130 [ksocklnd]
Jan 22 15:45:19 pdat0103 kernel: [<ffffffffa07d3531>] ? ksocknal_process_receive+0x2b1/0xa00 [ksocklnd]
Jan 22 15:45:19 pdat0103 kernel: [<ffffffffa07d6145>] ? ksocknal_scheduler+0x105/0x760 [ksocklnd]
Jan 22 15:45:19 pdat0103 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 22 15:45:19 pdat0103 kernel: [<ffffffffa07d6040>] ? ksocknal_scheduler+0x0/0x760 [ksocklnd]
Jan 22 15:45:19 pdat0103 kernel: [<ffffffff81096a36>] ? kthread+0x96/0xa0
Jan 22 15:45:19 pdat0103 kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Jan 22 15:45:19 pdat0103 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 22 15:45:19 pdat0103 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 22 15:45:19 pdat0103 kernel: --[ end trace 72e77805097e441d ]--
Jan 22 15:45:19 pdat0103 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
Jan 22 11:10:21 pdat0122 kernel: -----------[ cut here ]-----------
Jan 22 11:10:21 pdat0122 kernel: WARNING: at lib/list_debug.c:51 list_del+0x8d/0xa0() (Not tainted)
Jan 22 11:10:21 pdat0122 kernel: Hardware name: empty
Jan 22 11:10:21 pdat0122 kernel: list_del corruption. next->prev should be ffff88168fc7b9c0, but was ffff881c350b68ac
Jan 22 11:10:21 pdat0122 kernel: Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl osp(U) ofd(U) lfsck(U) ost(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U)
lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) mpt2sas scsi_transport_sas raid
_class mptctl mptbase autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bonding 8021q garp stp llc ipv6 sg ses enclosure igb ptp pps_core microcode serio_raw i2c_i801 i
2c_core iTCO_wdt iTCO_vendor_support e1000e ioatdma dca i7core_edac edac_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif megaraid_sas ahci dm_mirror dm_region_hash dm_log dm_mod [l
ast unloaded: scsi_wait_scan]
Jan 22 11:10:21 pdat0122 kernel: Pid: 14852, comm: ll_ost01_014 Not tainted 2.6.32-358.18.1.el6_lustre.x86_64 #1
Jan 22 11:10:21 pdat0122 kernel: Call Trace:
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff8106e3e7>] ? warn_slowpath_common+0x87/0xc0
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff8106e4d6>] ? warn_slowpath_fmt+0x46/0x50
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff812896cd>] ? list_del+0x8d/0xa0
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff81167595>] ? cache_alloc_refill+0x145/0x240
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff81167fec>] ? kmem_cache_alloc_trace+0x17c/0x1b0
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa04746ce>] ? lu_context_init+0x4e/0x240 [obdclass]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0abd3e6>] ? ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0b716de>] ? osd_trans_start+0x20e/0x670 [osd_ldiskfs]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0c7a39d>] ? ofd_trans_start+0x22d/0x3f0 [ofd]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0c7e15c>] ? ofd_attr_set+0x38c/0x6c0 [ofd]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0c6fd28>] ? ofd_setattr+0x678/0xc10 [ofd]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa05f7f9e>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0c201bb>] ? ost_setattr+0x30b/0x930 [ost]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0c231bd>] ? ost_handle+0x1f8d/0x44d0 [ost]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff81065463>] ? dequeue_entity+0x113/0x2e0
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff81057310>] ? __dequeue_entity+0x30/0x50
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa05ff9cb>] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0607e25>] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa03374ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa034827f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa05ff4c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff81063410>] ? default_wake_function+0x0/0x20
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa060918d>] ? ptlrpc_main+0xaed/0x1740 [ptlrpc]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa06086a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff81096a36>] ? kthread+0x96/0xa0
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 22 11:10:21 pdat0122 kernel: --[ end trace e4af010803407075 ]--
Jan 22 11:10:21 pdat0122 kernel: -----------[ cut here ]-----------
Jan 22 11:10:21 pdat0122 kernel: WARNING: at lib/list_debug.c:51 list_del+0x8d/0xa0() (Tainted: G W --------------- )
Jan 22 11:10:21 pdat0122 kernel: Hardware name: empty
Jan 22 11:10:21 pdat0122 kernel: list_del corruption. next->prev should be ffff881139f28640, but was 976a2e4410a82244
Jan 22 11:10:21 pdat0122 kernel: Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl osp(U) ofd(U) lfsck(U) ost(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bonding 8021q garp stp llc ipv6 sg ses enclosure igb ptp pps_core microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support e1000e ioatdma dca i7core_edac edac_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif megaraid_sas ahci dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Jan 22 11:10:21 pdat0122 kernel: Pid: 14852, comm: ll_ost01_014 Tainted: G W --------------- 2.6.32-358.18.1.el6_lustre.x86_64 #1
Jan 22 11:10:21 pdat0122 kernel: Call Trace:
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff8106e3e7>] ? warn_slowpath_common+0x87/0xc0
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff8106e4d6>] ? warn_slowpath_fmt+0x46/0x50
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff812896cd>] ? list_del+0x8d/0xa0
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff81167595>] ? cache_alloc_refill+0x145/0x240
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff81167fec>] ? kmem_cache_alloc_trace+0x17c/0x1b0
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa04746ce>] ? lu_context_init+0x4e/0x240 [obdclass]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0abd3e6>] ? ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0b716de>] ? osd_trans_start+0x20e/0x670 [osd_ldiskfs]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0c7a39d>] ? ofd_trans_start+0x22d/0x3f0 [ofd]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0c7e15c>] ? ofd_attr_set+0x38c/0x6c0 [ofd]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0c6fd28>] ? ofd_setattr+0x678/0xc10 [ofd]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa05f7f9e>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0c201bb>] ? ost_setattr+0x30b/0x930 [ost]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0c231bd>] ? ost_handle+0x1f8d/0x44d0 [ost]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff81065463>] ? dequeue_entity+0x113/0x2e0
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff81057310>] ? __dequeue_entity+0x30/0x50
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa05ff9cb>] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa0607e25>] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa03374ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa034827f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa05ff4c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff81063410>] ? default_wake_function+0x0/0x20
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa060918d>] ? ptlrpc_main+0xaed/0x1740 [ptlrpc]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffffa06086a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff81096a36>] ? kthread+0x96/0xa0
Jan 22 11:10:21 pdat0122 kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20